illusionist
12-18-2002, 03:10 PM
i got this error for my sites not being indexed by the search engines.... ERROR_DISALLOW: Crawling Disallowed By robots.txt
Below is my robots.txt file....can anybody spot the problem?
# Robots.txt file created by http://www.stickysauce.com
# For domain: http://www.iomx.com
# All robots will spider the domain
User-agent: *
Disallow:
# Disallow *
User-agent: *
Disallow: /
# Disallow *
User-agent: *
Disallow: /
JSpired
12-18-2002, 05:18 PM
This (http://www.searchengineworld.com/robots/robots_tutorial.htm) might help.
MarkIL
12-18-2002, 05:57 PM
Um, that file will DISALLOW all robots. That means that any conforming spider will NOT CRAWL your page.
http://www.robotstxt.org/wc/exclusion-admin.html
illusionist
12-18-2002, 06:09 PM
yes i i know......:bawling: thanx for the help guys...i corrected the error....:rolleyes:
Yep.. these lines:
User-agent: *
Disallow: /
... say that you're disallowing all robots (the wildcard * ) from crawling the root of your site. And you're saying it a couple of times! :)
Best bet, if you aren't sure you'll have it right is just to get rid of the file. You really don't need one if you want to allow all crawlers to go everywhere.
That said, that's probably a bad idea... there are some that you may as well try to exclude, and a lot of sites have content that you definitely don't want to have show up in a search engine.
illusionist
12-18-2002, 06:17 PM
how is this, is this right?
# Robots.txt file created by http://www.stickysauce.com
# For domain: http://www.iomx.com
# All robots will spider the domain
User-agent: *
Disallow:
