Web Hosting Talk : Web Hosting Main Forums : Web Hosting : Ok can someone please tell me who this crawl bot belongs to?

[thomor25]

I think it might be google but I'm not sure, its the top one.

This is from awstatsm first is teh name of the bot, second is the times accessing the site, 3rd is bandwidth used, and 4th is the date accessed.


Unknown robot (identified by 'crawl') 255 1.44 MB 15 Mar 2003 - 04:36
Googlebot (Google) 188 3.00 MB 16 Mar 2003 - 10:43
WISENutbot (Looksmart) 139 2.22 MB 17 Mar 2003 - 06:47
Inktomi Slurp 43 574.99 KB 17 Mar 2003 - 05:39
Unknown robot (identified by 'robot') 2 45.00 KB 06 Mar 2003 - 03:09
Netcraft Web Server Survey 1 0 Bytes 14 Mar 2003 - 23:22

[UH-Matt]

Well the googlebot one is google duh!

The Unknown one could be from many different sources. Theres too many bots around these days.

[thomor25]

Originally posted by UH-Matt
Well the googlebot one is google duh!

The Unknown one could be from many different sources. Theres too many bots around these days.

well yeah duh .... but i heard alot of people here talking about the google bot doing a "deep crawl" and I wondered if the "crawl" meant that it was google cause it had been accessing the site alot.

[eddy2099]

Probably it is some other web crawlers. I've seen it in my awstats but do not know where it is from. I have not check the raw logs, perhaps it would provide a little more info such as IP address ? With which, you could do a trace back ?

[sprintserve]

If it is from Google, it will be identified as Googlebot. so the unknown bot... is... well unknown. If you really want to, perhaps you can find out the ip, and check it out on arin.

[JayC]

Yep, anything from Google is identified... as is any crawler from a legitimate search engine. But there are all kinds of crawls done for any number of purposes. If you want to try to figure out who they are, the raw logs would be the way to identify who the IP address belongs to. Still that may won't tell you much.

Those two "unknown" listings, by the way, actually could be more than one crawler lumped together, I'd guess (I don't use awstats). It looks like anything that identifies itself as 'crawl' is in the first one and anything that identifies itself as 'spider' is in that one.

[thomor25]

Originally posted by JayC
Yep, anything from Google is identified... as is any crawler from a legitimate search engine. But there are all kinds of crawls done for any number of purposes. If you want to try to figure out who they are, the raw logs would be the way to identify who the IP address belongs to. Still that may won't tell you much.

Those two "unknown" listings, by the way, actually could be more than one crawler lumped together, I'd guess (I don't use awstats). It looks like anything that identifies itself as 'crawl' is in the first one and anything that identifies itself as 'spider' is in that one.

well the only ips that accessed my site at that time was from www.ripe.net

[sprintserve]

That means... that the bots are from Europe (thus Arin will report the ips as belonging to Ripe). Go to Ripe and try the same IPs again.

[thomor25]

did what you said and it gave me this

inetnum: 80.8.54.0 - 80.8.72.255
netname: FR-FT-WIC
descr: France Telecom Wanadoo Interactive Cable
descr: bas-1.sqy.net
country: FR
admin-c: WICT1-RIPE
tech-c: WICT1-RIPE
status: ASSIGNED PA
remarks: for hacking, spamming or security problems send ALSO mail to
remarks: abuse@cablewanadoo.com
remarks: for ANY problem send mail to gestionip.ft@francetelecom.com
notify: gestionip.ft@francetelecom.com
mnt-by: FT-BRX
changed: gestionip.ft@francetelecom.com 20011002
source: RIPE

route: 80.8.0.0/16
descr: France Telecom
descr: Wanadoo Interactive Cable
remarks: -------------------------------------------
remarks: For Hacking, Spamming or Security problems
remarks: send mail to abuse@cablewanadoo.com ONLY
remarks: -------------------------------------------
origin: AS3215
mnt-by: RAIN-TRANSPAC
mnt-by: FT-BRX
changed: karim@rain.fr 20010612
changed: karim@rain.fr 20020130
changed: gestionip.ft@francetelecom.com 20020909
source: RIPE

role: Wanadoo Interactive Cable Technical Role
address: France Telecom Wanadoo Interactive Cable
address: 40, rue Gabriel Crié
address: 92240 Malakoff
address: FR
phone: +33 1 58 88 54 16
e-mail: abuse@cablewanadoo.com
admin-c: MM2888-RIPE
tech-c: ML16648-RIPE
nic-hdl: WICT1-RIPE
mnt-by: FT-BRX
changed: gestionip.ft@francetelecom.com 20010517
changed: gestionip.ft@francetelecom.com 20020531
source: RIPE

[sprintserve]

Well.. Wanadoo is an ISP in France... So whoever running that bot is running it from their home cable, Unless the bot did something illegal, that's as far as you can go We can only speculate why they are crawling your site.