Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : robots.txt help. Need semi explanation

[maxpower097]

Ok I'm a little new to the whole robots.txt file. I have been reading up on them and some sites say that its to block files from the crawlers, and tell crawlers to scan certain area's. But on other sites their saying its only to block crawlers. Which is it? Should I only use it to block files? Or is there a way to index my site in their to make sure it gets crawled? Every time I try to validate one it says it MUST HAVE A DISSALLOW statement.

Any help would be appreciated.
thanks
maxpower

[Biju]

http://www.eulagam.com/search-engine...obotstxt-file/

This tutorial might help you.

[Domainitor]

Do yourself a favor: don't list any directories or files that are not referenced by the site. Doing so causes web scrapers to pick through them. The well-behaved spiders from the various search engines will leave them alone, but listing them in your robots.txt will cause ill-behaved spiders to scrounge through your stuff.