Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : Problem - Robots.txt appearing in SEPRs

[webaholic]

Hi,

I have seen recently robots.txt for my site & one from my blog (Blogger custom domain) is appearing in SERPs (snapshots attached). I have no idea how did this happen. I have never placed the robots.txt link to any forum or other sites. Whereas I often use these free online website analysis tools like websitegrader.com/woorank.com which may checks robots file while analyzing.

I am not sure, but I hope using url removal tool in Google Webmaster Tools won't be a good idea. What I can do to fix this? Do I really need to care about this?

Thanks,
Jim

[lynxus]

What on earth were you searching for?

I wouldn't care tbh..
Pretty much every site has that file, so its no secret that its there ( you wont need to link to it as google looks for it anyway )

Why its being indexed is odd mind you.. But IMO it doesnt really matter.

[webaholic]

Originally Posted by lynxus

What on earth were you searching for?

Hope you have probably came across this "site:yourdomain.com". It tells you how many pages of yours is indexed by Google. This is something on earth which almost every webmaster wants to know.

[lynxus]

I only asked because I tried that, along with the filetype tags to narrow my search down however robots.txt doesn't appear for any domain i look at.

Very strange.

id personally raise the question on googles own forums.. You have more of a chance of getting it fixed if it is a bug. ( ive had many things tweaked by admins due to issues with GWT and other systems they run.. )

[sijugk]

Originally Posted by webaholic

Hi,

I have seen recently robots.txt for my site & one from my blog (Blogger custom domain) is appearing in SERPs (snapshots attached) Do I really need to care about this?

Thanks,
Jim

Strange!!! I do not think you really need to care it. If I am you, I won't take any effort to remove it from google index list.

[jonathan cole]

There is no any problem if it has been index by search engine let it be. If you have placed some thing in your directory it must be indexed.

[nicoweb]

I have never seen that before just make sure that there are no links pointing to the robots.txt file and your robots.txt hasn't been included in your sitemap by accident.