Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : Advice on robots.txt

[pctank]

Please give advise about adding a robots.txt file to your website directory and whats the perpose and do you need it?

Wow im short of breath

[Sam Robertson]

This gives info on robots.txt:
http://www.robotstxt.org/robotstxt.html

[angilina]

I think you need to add it and its really easy.

Just make a file and name it "robot.txt"

and put this in it

User-agent: *
Disallow:

This will tell search bot to index all your pages.

[Feydakin]

You never have to tell the search engines to index your pages, only what pages to not index..

[nuclei]

Originally Posted by angilina

Just make a file and name it "robot.txt"

#1. It is "robots.txt" NOT "robot.txt".

Originally Posted by angilina

User-agent: *
Disallow:

This will tell search bot to index all your pages.

Actually if no param is given, as in your example,after the disallow, it probably will tell the bots to NOT index any of your pages as the default is * if I recall correctly.

So if this user used your advice, he would have lost any pages in the engines that he had already.

Please read, learn, and not give advice unless you know what you are talking about.

[nuclei]

Originally Posted by pctank

Please give advise about adding a robots.txt file to your website directory and whats the perpose and do you need it

You do not need it unless you actually wish to block the spiders from certain files or directories on your web site. I would suggest uploading an empty robots.txt file to your main html directory anyways, to cut down on the filesize of your apache error log due to it not being found at all.

And the web site that Sam gave you above is a good place to learn if you actually DO want to block things from the spiders.

[pctank]

Thanks that is a good idea

Thanks guys

[Biju]

Here is a article

http://www.seopapers.com/article/357

Well the author is none other than me, self promotion. LOL

[intesync]

Hi Isak,

It's best used to prevent search engines from indexing pages and directories that you don't want to be displayed to the public. It's not just Google and Yahoo! There are free/paid search engine services that anyone can use to search through your sites for information. PicoSearch is a quick example.

Another reason would be to keep your site from archived by the Internet Archive (or the Internet Time Machine). Not too many people know this but it can be bad for publicity if you have a unsuitable page that you cannot removed from the Internet.

[Zafar Ahmed]

Okay, the purpose behind the robot file is to guide the robot. If you let it like that it will index every page unless you stop him from doing it do or stop him from indexing a specific link.

Other than that, people use it for sitemaps as well. If you don't have robot.txt; you don't need to worry if Google is going to index you or not. The Algo is different now.

[pctank]

Originally Posted by nuclei

You do not need it unless you actually wish to block the spiders from certain files or directories on your web site. I would suggest uploading an empty robots.txt file to your main html directory anyways, to cut down on the filesize of your apache error log due to it not being found at all.

And the web site that Sam gave you above is a good place to learn if you actually DO want to block things from the spiders.

Ok shall i apply a empty robot.txt file into my directory just for the hell of it????

[Feydakin]

You can.. It won't matter one way or the other.. I do that with some sites just to stop the 404 error reports on that domain..

[angilina]

Originally Posted by nuclei

#1. It is "robots.txt" NOT "robot.txt".



Actually if no param is given, as in your example,after the disallow, it probably will tell the bots to NOT index any of your pages as the default is * if I recall correctly.

So if this user used your advice, he would have lost any pages in the engines that he had already.

Please read, learn, and not give advice unless you know what you are talking about.

Take a look at this page and try to learn

robotstxt.org/robotstxt.html

there is a difference between to codes

To exclude all robots from the entire server

User-agent: *
Disallow: /


To allow all robots complete access

User-agent: *
Disallow:


May be you forgot to wear your glasses

[Biju]

Robots.txt is neccessary if you are looking to block your seayorch engine. Its the way you guide your bots when entering into your websites.

[Zafar Ahmed]

Originally Posted by Biju

Robots.txt is neccessary if you are looking to block your seayorch engine. Its the way you guide your bots when entering into your websites.

Biju - do you guys call "search" "seayorch" down there in India?