hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : Advice on robots.txt
Reply

Forum Jump

Advice on robots.txt

Reply Post New Thread In SEO / SEM  Discussions Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 03-21-2008, 05:13 PM
pctank pctank is offline
WHT Addict
 
Join Date: Mar 2008
Posts: 102
Exclamation

Advice on robots.txt


Please give advise about adding a robots.txt file to your website directory and whats the perpose and do you need it?

Wow im short of breath

__________________
We Bring Good Things to Life!
www.pctank.co.uk
- Web Design - Search Engine Optimization - Graphic Design -




Sponsored Links
  #2  
Old 03-21-2008, 05:16 PM
Sam Robertson Sam Robertson is offline
Aspiring Evangelist
 
Join Date: Jan 2008
Location: United Kingdom
Posts: 408
This gives info on robots.txt:
http://www.robotstxt.org/robotstxt.html

  #3  
Old 03-21-2008, 06:29 PM
angilina angilina is offline
Newbie
 
Join Date: Mar 2008
Posts: 27
I think you need to add it and its really easy.

Just make a file and name it "robot.txt"

and put this in it

User-agent: *
Disallow:

This will tell search bot to index all your pages.

Sponsored Links
  #4  
Old 03-21-2008, 08:41 PM
Feydakin Feydakin is offline
Junior Guru Wannabe
 
Join Date: Feb 2008
Posts: 74
You never have to tell the search engines to index your pages, only what pages to not index..

__________________
Steve
Metal Monster Marketing : Internet Marketing

  #5  
Old 03-21-2008, 08:47 PM
nuclei nuclei is offline
WHT Addict
 
Join Date: Jan 2002
Posts: 159
Quote:
Originally Posted by angilina View Post
Just make a file and name it "robot.txt"
#1. It is "robots.txt" NOT "robot.txt".

Quote:
Originally Posted by angilina View Post
User-agent: *
Disallow:

This will tell search bot to index all your pages.
Actually if no param is given, as in your example,after the disallow, it probably will tell the bots to NOT index any of your pages as the default is * if I recall correctly.

So if this user used your advice, he would have lost any pages in the engines that he had already.

Please read, learn, and not give advice unless you know what you are talking about.

__________________
William Cross
Don Halbert *play site*
william@seofox.com

  #6  
Old 03-21-2008, 08:49 PM
nuclei nuclei is offline
WHT Addict
 
Join Date: Jan 2002
Posts: 159
Quote:
Originally Posted by pctank View Post
Please give advise about adding a robots.txt file to your website directory and whats the perpose and do you need it
You do not need it unless you actually wish to block the spiders from certain files or directories on your web site. I would suggest uploading an empty robots.txt file to your main html directory anyways, to cut down on the filesize of your apache error log due to it not being found at all.

And the web site that Sam gave you above is a good place to learn if you actually DO want to block things from the spiders.

__________________
William Cross
Don Halbert *play site*
william@seofox.com

  #7  
Old 03-22-2008, 03:52 AM
pctank pctank is offline
WHT Addict
 
Join Date: Mar 2008
Posts: 102
Thanks that is a good idea

Thanks guys

__________________
We Bring Good Things to Life!
www.pctank.co.uk
- Web Design - Search Engine Optimization - Graphic Design -


  #8  
Old 03-22-2008, 06:32 AM
Biju Biju is offline
Big fan of RajiniKanth!!!
 
Join Date: Sep 2004
Location: Chennai , India
Posts: 4,592
Here is a article

http://www.seopapers.com/article/357

Well the author is none other than me, self promotion. LOL

  #9  
Old 03-22-2008, 09:33 PM
intesync intesync is offline
Newbie
 
Join Date: Mar 2008
Location: Silicon Valley
Posts: 5
Hi Isak,

It's best used to prevent search engines from indexing pages and directories that you don't want to be displayed to the public. It's not just Google and Yahoo! There are free/paid search engine services that anyone can use to search through your sites for information. PicoSearch is a quick example.

Another reason would be to keep your site from archived by the Internet Archive (or the Internet Time Machine). Not too many people know this but it can be bad for publicity if you have a unsuitable page that you cannot removed from the Internet.

  #10  
Old 03-23-2008, 01:21 AM
Zafar Ahmed Zafar Ahmed is offline
Web Hosting Master
 
Join Date: Mar 2004
Location: Pakistan
Posts: 2,752
Okay, the purpose behind the robot file is to guide the robot. If you let it like that it will index every page unless you stop him from doing it do or stop him from indexing a specific link.

Other than that, people use it for sitemaps as well. If you don't have robot.txt; you don't need to worry if Google is going to index you or not. The Algo is different now.

__________________
I'm Zafar Ahmed.
I provide
SEO Services & eMarketing consultancy
I'll be glad to hear from you


  #11  
Old 03-23-2008, 03:11 PM
pctank pctank is offline
WHT Addict
 
Join Date: Mar 2008
Posts: 102
Quote:
Originally Posted by nuclei View Post
You do not need it unless you actually wish to block the spiders from certain files or directories on your web site. I would suggest uploading an empty robots.txt file to your main html directory anyways, to cut down on the filesize of your apache error log due to it not being found at all.

And the web site that Sam gave you above is a good place to learn if you actually DO want to block things from the spiders.
Ok shall i apply a empty robot.txt file into my directory just for the hell of it????

__________________
We Bring Good Things to Life!
www.pctank.co.uk
- Web Design - Search Engine Optimization - Graphic Design -


  #12  
Old 03-23-2008, 03:30 PM
Feydakin Feydakin is offline
Junior Guru Wannabe
 
Join Date: Feb 2008
Posts: 74
You can.. It won't matter one way or the other.. I do that with some sites just to stop the 404 error reports on that domain..

__________________
Steve
Metal Monster Marketing : Internet Marketing

  #13  
Old 03-23-2008, 04:28 PM
angilina angilina is offline
Newbie
 
Join Date: Mar 2008
Posts: 27
Quote:
Originally Posted by nuclei View Post
#1. It is "robots.txt" NOT "robot.txt".



Actually if no param is given, as in your example,after the disallow, it probably will tell the bots to NOT index any of your pages as the default is * if I recall correctly.

So if this user used your advice, he would have lost any pages in the engines that he had already.

Please read, learn, and not give advice unless you know what you are talking about.
Take a look at this page and try to learn

robotstxt.org/robotstxt.html

there is a difference between to codes

To exclude all robots from the entire server

User-agent: *
Disallow: /


To allow all robots complete access

User-agent: *
Disallow:


May be you forgot to wear your glasses

  #14  
Old 03-23-2008, 04:32 PM
Biju Biju is offline
Big fan of RajiniKanth!!!
 
Join Date: Sep 2004
Location: Chennai , India
Posts: 4,592
Robots.txt is neccessary if you are looking to block your seayorch engine. Its the way you guide your bots when entering into your websites.

  #15  
Old 03-23-2008, 05:20 PM
Zafar Ahmed Zafar Ahmed is offline
Web Hosting Master
 
Join Date: Mar 2004
Location: Pakistan
Posts: 2,752
Quote:
Originally Posted by Biju View Post
Robots.txt is neccessary if you are looking to block your seayorch engine. Its the way you guide your bots when entering into your websites.
Biju - do you guys call "search" "seayorch" down there in India?

__________________
I'm Zafar Ahmed.
I provide
SEO Services & eMarketing consultancy
I'll be glad to hear from you


Reply

Related posts from TheWhir.com
Title Type Date Posted
Rackspace Launches One-Stop Digital Services Practice Web Hosting News 2014-03-03 16:14:03
Nominet Pilot Program Hopes to Help SMBs Combat Cyber Threats with Support, Advice Web Hosting News 2014-02-07 15:12:39
Easyspace and Crunch Partner to Package Hosting and Accounting Services Web Hosting News 2013-07-15 15:03:55
WHD.global 2013: OnApp Shares Cloud Best Practices through New Website Web Hosting News 2013-03-19 07:33:08
Why Security is Key to the Growth of the Cloud Business with Gray Hall of Alert Logic Web Hosting News 2012-09-20 12:11:01


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?