Porky
05-28-2004, 11:43 PM
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /board/skin/style_images/LiteBar--563/
thats going to stop all the bots from spidering my pictures right?
yes, that would work but remember not all bots obey robots.txt, the majority do but some of the smaller search engines ignore it.
Kronikas
05-29-2004, 05:10 AM
What are exactly the meaning of robots.txt file?
saul alvarado
It's a protocol that allows you to specify which areas of your site search engine bots can and cannot access, and also to allow or deny different bots.
Most search engine bots will look for the file robots.txt when they first come to your site before they start to index the pages.
http://www.robotstxt.org/wc/robots.html
stripeyteapot
05-29-2004, 11:20 AM
Remember, just because it's in the robots.txt file not to search, doesn't mean people cannot find them :)
ktwilight
05-29-2004, 01:43 PM
this might be slightly offtopic...
what are the other ways people can find files that aren't made public? eg php includes
stripeyteapot
05-29-2004, 04:55 PM
Anyone can view the robots.txt file. If you include a private file they will also be able to view it by folloiwng the correct url. They can also view the directory if no index file is set.
Using SSI this simple includes the file into the source code, so they don't even have to follow the URL, it's all in the page. There are ways to protect your code, but some sad kid with no friends and lots of spots will always hack through it.