sitekeeper
08-29-2002, 04:30 AM
Is anyone familiar with Googles Googlebot and how long it spends on a site? I was of the understanding that google only indexed one level deep.
These 2 bots have been on my site for over 25 minutes now.
216.239.46.20 crawl1.googlebot.com
64.68.82.30 crawler10.googlebot.com
I have a MySQL/PHP driven website with a 2MB database and about 25mb of html. The html is all 3 levels down from the index page.
Acronym BOY
08-29-2002, 04:49 AM
Youll find the information in this thread on how google works, how google deals with SIDs (or rather how it doesnt), internal linkings, and how it can be fooled with "static content" useful:
http://www.webhostingtalk.com/showthread.php?s=&threadid=69200
This answered most of my questions on the Googlebot:
http://www.google.com/bot.html
Originally posted by sitekeeper
I was of the understanding that google only indexed one level deep. That understanding is incorrect. But it does appear that the current crawl is more extensive than most.
pgrote
08-29-2002, 02:19 PM
This crawl is amazing.
I haven't seen them go this deep, for so long and so many times in a long time.
sitekeeper
08-29-2002, 02:42 PM
Originally posted by pgrote
This crawl is amazing.
I haven't seen them go this deep, for so long and so many times in a long time.
My thoughts exactly, if I didn’t know better I would think they stopped by to rest. :)
Well hopefully I will at least show in Google. :cartman:
One of the bots left after 30 minutes and the other after 35 minutes.
