07-16-2003, 05:12 PM
|
|
Registered User
|
|
Join Date: Oct 2002
Posts: 292
|
|
I have had 10 Inktomi bots on my site for 2 days now. What the hell could they be doing? What is also weird is that some posted on my forum from the host name 'inktomi1-cdf.server.ntl.com.'
|
07-16-2003, 05:24 PM
|
|
Community Leader
|
|
Join Date: Oct 2002
Location: cognito
Posts: 17,329
|
|
Inktomi has recently implemented a method to follow and index dynamic pages (PHP and so on). Does the site use PHP?
They claim slurp (the bot) will honor robots txt exclusion files.
After it exhausts all the possible links on your site, it will leave...maybe. I had one caught in a shopping cart, and it was buying items. Left without paying, though...
|
07-16-2003, 05:29 PM
|
|
New Member
|
|
Join Date: Nov 2002
Location: Roseville, CA
Posts: 1
|
|
I've been getting that a lot on my forum. It's Inktomi's indexing bot, "Slurp" and it's indexing your site to be included in the search database.
|
07-16-2003, 05:46 PM
|
|
Junior Guru Wannabe
|
|
Join Date: May 2003
Location: Overland Park, KS
Posts: 45
|
|
We have had over 40 of these friggin "Bots" for 7 days now. Used up about 3GB of bandwidth so far...
__________________
KCRadio.net - The source for KC Radio news
|
07-16-2003, 05:54 PM
|
|
Web Hosting Master
|
|
Join Date: May 2003
Location: Philadelphia
Posts: 968
|
|
Quote:
Originally posted by HostPlains
We have had over 40 of these friggin "Bots" for 7 days now. Used up about 3GB of bandwidth so far...
|
robots.txt is your friend, if you dont want the bots there tell them not to come back.
__________________
http://www.eBoundary.com - Let us help you expand your eBoundaries!
Fast, Secure and reliable FreeBSD shared, reseller and dedicated hosting.
FREE Peace of mind with every account!
|
07-16-2003, 06:40 PM
|
|
Web Hosting Master
|
|
Join Date: Sep 2002
Location: Knoxville, TN
Posts: 645
|
|
Quote:
Originally posted by bear
Inktomi has recently implemented a method to follow and index dynamic pages (PHP and so on). Does the site use PHP?
They claim slurp (the bot) will honor robots txt exclusion files.
After it exhausts all the possible links on your site, it will leave...maybe. I had one caught in a shopping cart, and it was buying items. Left without paying, though...
|
Just curious -- what does a bot buy?
__________________
Laura K.
http://www.madmousergraphics.com
graphic design for grownups
|
07-16-2003, 06:44 PM
|
|
Web Hosting Master
|
|
Join Date: Oct 2002
Location: Virginia
Posts: 787
|
|
Quote:
Originally posted by bear
After it exhausts all the possible links on your site, it will leave...maybe.
|
on one of my sites it has been going through my forum for over a week using 268MB
|
07-16-2003, 06:46 PM
|
|
Web Hosting Master
|
|
Join Date: Jul 2002
Posts: 1,441
|
|
common feed the bots 
|
07-16-2003, 07:17 PM
|
|
Web Hosting Master
|
|
Join Date: Apr 2003
Posts: 634
|
|
shouldnt your site be lised on http://www.inktomi.com after it spidered you? mine is getting spidered for like a month now and it still havent been listed
|
07-16-2003, 07:37 PM
|
|
Web Hosting Master
|
|
Join Date: Jul 2002
Location: UK
Posts: 2,026
|
|
lol...
they arent bots.. they are ntl proxies... everyone on the isp ntl has their traffic on port 80 routed through one(and ntl has a large % of the uk population)
that particular one is a user from cardiff in the uk.
__________________
Gone.
|
07-16-2003, 07:44 PM
|
|
Junior Guru Wannabe
|
|
Join Date: May 2001
Location: United Kingdom
Posts: 57
|
|
Inktomi.com is the corporate site - you should see your pages on MSN, HotBot and a few Infospace properties when Inktomi next updates.
Matrix 28,
Inktomi used to supply server software as well as having a web search product. NTL is one of their customers. That reference on your forum ('inktomi1-cdf.server.ntl.com) is to the server software product.
__________________
search engine positioning and ranking services. Guaranteed results. http://www.1design4life.co.uk
|
07-16-2003, 09:31 PM
|
|
Web Hosting Evangelist
|
|
Join Date: Feb 2003
Location: United Kingdom
Posts: 522
|
|
Just had the same thing on our forums.
I thought wow, this is busy - after checking out the IP they all belonged to Slurp though 
As for the NTL issue, NTL uses Inktomi's proxies - although this is unrelated to the original poster.
Seems as though it's all over now though
Kevin
|
07-17-2003, 12:06 AM
|
|
Retired ***ulator
|
|
Join Date: Jan 2003
Location: Let's see.....
Posts: 4,205
|
|
I've been seeing the same thing. Last night my forum's online thingy said there was 5 guests viewing the board. When I went into Admin mode, I noticed the IP belonged to Inktomi.
__________________
73's, Kim
Everything happens for a reason I make up.
|
07-17-2003, 12:35 AM
|
|
Web Hosting Guru
|
|
Join Date: May 2003
Posts: 282
|
|
I've had a similar situation on my forum except with Googlebot. It has literally been at my site for over 7 days straight.
I would ban it from the forum but I use adsense and wish to allow them to index for those purposes.
The good thing is, googlebot tends to play nice and so hasn't hit the server too hard or fast but nevertheless, it's been there grabbing pages for over a week (continuosly).
|
07-17-2003, 04:53 AM
|
|
Registered User
|
|
Join Date: Oct 2002
Posts: 292
|
|
Yes, it is a vBulletin, thanks for all the help guys.
|
| Thread Tools |
Search this Thread |
|
|
|
| Display Modes |
 Linear Mode
|
| Postbit Selector |
|
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
|
|
|
| Login: |
|
|
| Advertisement: |
|
|
| Web Hosting News: |
|
|
|
