Web Hosting Talk







View Full Version : How to setup robots.txt file to ban these bots?


illusionist
02-08-2003, 05:34 PM
I want to ban the following bots from all files on the server

psbot/0.1
(+http://www.picsearch.com/bot.html)


NPBot-1/2.0
(+http://www.picsearch.com/bot.html)

How do i setup the robots.txt file to ban these specific bots from scanning my site?

Currently its like this:
# For domain: http://www.iomx.com

# All robots will spider the domain
User-agent: *
Disallow:

MGCJerry
02-08-2003, 05:43 PM
From the provided link...


robots.txt example:
Include the following in your robots.txt file:

User-agent: psbot
Disallow: /


META tag example:
Include the following in the head section of your web-page:

<meta name="robots" content="noindex,nofollow">


Thanks for the heads up on this one... I don't want this one on my site either. Then again anyone who takes my images are competely lame. My stuff isn't worth taking. :eek:

illusionist
02-08-2003, 05:47 PM
no problem!

so the end result would look like this? im asking this cause i really dont want mess with robots file, i really had bad experince once..

# For domain: http://www.iomx.com

# All robots will spider the domain
User-agent: psbot
Disallow: /
User-agent: NPBot
Disallow: /

is the above correct?

illusionist
02-09-2003, 05:42 AM
??

MGCJerry
02-09-2003, 05:31 PM
Oops, I forgot about this post...

As far as I know from reading their site, all you need is:
User-agent: psbot
Disallow: /

But I don't think it would hurt anything to use what you have.