Robots.txt

Robots.txt is a file that contains information describing what directories a web-crawler (that conforms to the Robots Exclusion Standard) should not access or crawl.

Typically, you would list directories which a web-crawler should not access (and thus crawl). More information on the contents of a Robots.txt file and example rules can be found on Google's website.