I was just trawling through our web logs for September and was particularly interested by Google's spider: Last month Googlebot slurped (no pun intended) 812MB of our bandwidth.

A small price to pay for the traffic Google generates but it got me thinking about just how high Google's bandwidth bill must be each month.

To get a small idea of Google's bandwidth usage I thought it'd be interesting to see how much bandwidth Google used from web hosts alone.

If you don't mind looking at a summary of your logs from September add your figure to 812MB - let's see how quickly it gets up to 1TB :)

(Am I sad? Probably)

Kevin

googlebot used 353 KB of my site's bandwidth and it's increasing everyday ....:( :( pretty sad indeed


edited: should be KB not MB ...my bad ;)

last month googlebot used up 700 KB

that's very sad......

This month till now - 522.23KB
Last month - 1023.33KB

Still, the site ranks almost nothing.

Google uses terabytes a day for their spiders, based over thousands of servers.

Granted they can 'chew the juice' on your site for a full spider, but it's a good thing if they manage to fully catalog your website.

If it's a concern you can always place their bot in your robot.txt file and they'll avoid you in future.

Originally posted by cyberservers
This month till now - 522.23KB
Last month - 1023.33KB

Still, the site ranks almost nothing.

highly unlikely a site will rank under certain keyword phrases unless it takes into consideration factors of google's algorithm.

but your sites listed.. so googles done their job, now it's up to you.. to get it 'to rank' under keyword phrases.

Fast-Webcrawler (AllTheWeb) is just as bad on some sites i monitor, but ahh well.. if you dont want to be indexed into that particular search engine, you can always as akashik mentioned place it in your robots.txt
but i don't know many websites that dont want to have their website indexed into google and other major search engines.
if you think googles hitting your website too "hard" they mention;

"If you find that we are placing too high a load on your site, please let us know by sending us e-mail at googlebot@google.com"

Google used 9mb of my bandwidth so far this month.

Inktomi Slurp used 45mb last month...

This Month so far:
Googlebot (Google) 371 7.85 MB 07 Oct 2003 - 09:39
Unknown robot (identified by 'crawl') 294 1.74 MB 06 Oct 2003 - 23:05
Alexa (IA Archiver) 117 1.27 MB 07 Oct 2003 - 03:00
Fast-Webcrawler (AllTheWeb) 62 1.37 MB 07 Oct 2003 - 01:19
Scooter (AltaVista) 29 569.68 KB 06 Oct 2003 - 12:46
Inktomi Slurp 13 221.58 KB 06 Oct 2003 - 09:49
Unknown robot (identified by 'robot') 8 174.06 KB 04 Oct 2003 - 20:11
WISENutbot (Looksmart) 5 76.69 KB 07 Oct 2003 - 03:18
Jeeves 1 19.29 KB 01 Oct 2003 - 18:44
Voila 1 20.19 KB 01 Oct 2003 - 12:08

what is Unkown robot (identified by "crawl") this one seems to take the most bandwidth on mine.

Originally posted by SideFX
This Month so far:
Googlebot (Google) 371 7.85 MB 07 Oct 2003 - 09:39
Unknown robot (identified by 'crawl') 294 1.74 MB 06 Oct 2003 - 23:05
Alexa (IA Archiver) 117 1.27 MB 07 Oct 2003 - 03:00
Fast-Webcrawler (AllTheWeb) 62 1.37 MB 07 Oct 2003 - 01:19
Scooter (AltaVista) 29 569.68 KB 06 Oct 2003 - 12:46
Inktomi Slurp 13 221.58 KB 06 Oct 2003 - 09:49
Unknown robot (identified by 'robot') 8 174.06 KB 04 Oct 2003 - 20:11
WISENutbot (Looksmart) 5 76.69 KB 07 Oct 2003 - 03:18
Jeeves 1 19.29 KB 01 Oct 2003 - 18:44
Voila 1 20.19 KB 01 Oct 2003 - 12:08


Where and how did you get these information? :blush:

ML

I get that info from Awstats, or my Analog stats. If you use Cpanel.

Same here:)

Can we get that information for the whole server?

ML

I doubt it.

864.2 KB.
Not bad for some good ad's....lol

Heres mine:


Lycos 343 250.88 KB 02 Oct 2003 - 07:03
Googlebot (Google) 10 473.44 KB 06 Oct 2003 - 14:19
Inktomi Slurp 10 267.35 KB 07 Oct 2003 - 00:27
Jeeves 7 370.50 KB 06 Oct 2003 - 14:30
WISENutbot (Looksmart) 5 148.75 KB 06 Oct 2003 - 05:44
Unknown robot (identified by 'spider') 4 121.91 KB 06 Oct 2003 - 02:46
Unknown robot (identified by 'robot') 4 173.88 KB 05 Oct 2003 - 13:15
Scooter (AltaVista) 4 195.34 KB 06 Oct 2003 - 09:16
Unknown robot (identified by 'crawl') 2 115.50 KB 02 Oct 2003 - 13:18
Wget 1 37.81 KB 03 Oct 2003 - 06:07

Originally posted by nmluan
Where and how did you get these information? :blush:

ML

AwStats I presume?

My Stats for October so far:
3163 KB crawl22.googlebot.com
2370 KB crawl23.googlebot.com
964 KB crawl24.googlebot.com

Well, seems like Google needs to pay for their bandwidth usage. :cool:

Just curious, does the googlebot accept gzipped data?

http://www.webmasterworld.com/forum3/13390.htm

GoogleGuy snippet:

"I think someone here did a test and was surprised at how many web servers don't serve compressed pages correctly (like the early days of virtual hosting). So I wouldn't expect this anytime soon, but I agree in principle that this would be a good feature in the future. "

Google munched down almost 100 MB for us.

I hope they use 1 GB next month, I love Googlebot. :D

It wouldnt supprise me if they "slurped" up 1Tb/hour. Do you have any Idea as to what kind of server setup Google has?

If I remember correctly, Google has a 50,000 server Beowulf cluster that powers their engine. Yes, Fifty Thousand. At least that is the number I remember reading somehwere.


Lemme go double check that.

My bad, its only 4000 servers big. I don't know how I got to thinking it was 50,000.... must have been a post/comment on slashdot when I originally read the article.

Here is an article detailing the cluster: http://www.internetwk.com/lead/lead060100.htm

Originally posted by amish_geek
My bad, its only 4000 servers big. That article's pretty old. I think the last number I read was 8,000. And they're still growing.

for my website, Gawaher.com (http://gawaher.com) , googlebot consumes a daily average of: 42.85MB, or 1.285GB/month. Perhaps its high because they cache all my pages, updating too frequesntly.

ps. forgot to mention that perhaps also its because my website has around 25000 entries (http://www.google.com/search?q=site:gawaher.com+gawaher&num=100&hl=en&lr=&ie=UTF-8&filter=0) in google indexing, and a NO.1 ranking in my keyword of interest (http://www.google.com/search?num=100&hl=en&lr=&ie=ISO-8859-1&q=islamic+web+hosting) . How many entries you guys have for your websites?

This month so far:

Googlebot (Google) 364 7.52 MB

Last month:

Googlebot (Google) 2907 74.34 MB

I think Google was on our website almost everyday last month.

Unknown robot (identified by 'crawl') 1484 7.10 MB 26 Oct 2004 - 05:42
MSNBot 1166 28.93 MB 13 Oct 2004 - 23:50
Googlebot 888 9.65 MB 25 Oct 2004 - 23:13
Inktomi Slurp 789 4.22 MB 26 Oct 2004 - 03:45
ibm 523 1.11 MB 13 Oct 2004 - 11:14
MSIECrawler 392 713.04 KB 13 Oct 2004 - 08:37
Alexa (IA Archiver) 49 882.98 KB 25 Oct 2004 - 19:32
Unknown robot (identified by 'spider') 40 1.15 MB 26 Oct 2004 - 05:41
EchO! 23 29.28 KB 15 Oct 2004 - 17:00
BaiDuSpider 22 651.75 KB 13 Oct 2004 - 02:00
AskJeeves 19 587.06 KB 24 Oct 2004 - 10:45
LinkWalker 7 132.38 KB 05 Oct 2004 - 14:43
Unknown robot (identified by 'robot') 7 133.26 KB 23 Oct 2004 - 03:35
ExactSeek Crawler 6 146.02 KB 08 Oct 2004 - 06:54
eit 3 85.32 KB 03 Oct 2004 - 15:21
larbin 2 58.15 KB 02 Oct 2004 - 11:32
Yandex bot 2 26.46 KB 08 Oct 2004 - 05:29
CoolBot 1 28.44 KB 03 Oct 2004 - 15:20
InfoSeek Robot 1.0 1 28.34 KB 03 Oct 2004 - 15:20
SurveyBot 1 23.92 KB 11 Oct 2004 - 07:18
Walhello appie 1 38.75 KB 17 Oct 2004 - 17:27
Fish search 1 27.63 KB 09 Oct 2004 - 17:00
Netcraft 1 0 17 Oct 2004 - 15:10
cmc 1 6.21 KB 12 Oct 2004 - 09:12

anyone here working for google who could give us a sneakpeak at their stats? :D

I know in just one of google's dc space (one of their decent sized ones, and they have four) they have about 6 gigabit of connectivity.

craig is a family friend of ours :)

this month 643.26 MB

153.98 Meg for october...not that bad?