Results 1 to 8 of 8
  1. #1
    Join Date
    Apr 2003
    Posts
    634

    robots.txt. is this right?

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /images/
    Disallow: /board/skin/style_images/LiteBar--563/

    thats going to stop all the bots from spidering my pictures right?

  2. #2
    Join Date
    Apr 2003
    Location
    London, UK
    Posts
    4,695
    yes, that would work but remember not all bots obey robots.txt, the majority do but some of the smaller search engines ignore it.

  3. #3
    Join Date
    May 2004
    Location
    New Zealand
    Posts
    346
    yep thats good

  4. #4
    What are exactly the meaning of robots.txt file?

    saul alvarado

  5. #5
    Join Date
    Apr 2003
    Location
    London, UK
    Posts
    4,695
    It's a protocol that allows you to specify which areas of your site search engine bots can and cannot access, and also to allow or deny different bots.

    Most search engine bots will look for the file robots.txt when they first come to your site before they start to index the pages.

    http://www.robotstxt.org/wc/robots.html

  6. #6
    Remember, just because it's in the robots.txt file not to search, doesn't mean people cannot find them
    ServeYourSite
    Web hosting done right
    ██ Shared, Reseller and Dedicated web hosting
    An Easy Web Presence Company

  7. #7
    this might be slightly offtopic...

    what are the other ways people can find files that aren't made public? eg php includes

  8. #8
    Anyone can view the robots.txt file. If you include a private file they will also be able to view it by folloiwng the correct url. They can also view the directory if no index file is set.

    Using SSI this simple includes the file into the source code, so they don't even have to follow the URL, it's all in the page. There are ways to protect your code, but some sad kid with no friends and lots of spots will always hack through it.
    ServeYourSite
    Web hosting done right
    ██ Shared, Reseller and Dedicated web hosting
    An Easy Web Presence Company

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •