Web Hosting Talk







View Full Version : googlebot...30+ GB bandwidth per month?


tranceformer
12-31-2007, 01:07 PM
I recently checked my stats page and noticed an anomaly-googlebot is consuming about 30GB of bandwidth on my site per month.

if I wanted to block googlebot from www.mysite.com/directoryA/fileZ.php, how exactly would I do so? What would robots.txt look like, and where would I put robots.txt, in the directoryA directory?

Xeentech
12-31-2007, 01:56 PM
If it's just googlebot I'd setup a google site map.. That way you can tell it how often you expect the content on each page or area to be updated. Then you don't waste any bandwidth.

nnormal
01-02-2008, 01:59 PM
http://www.robotstxt.org/

http://www.sitemaps.org/protocol.php

serverman22
01-05-2008, 11:11 PM
wow thats a lot of bandwidth, make robot text file and put it in dir as files, i still cant belive it ate that much though amazing

wukas
02-03-2008, 07:53 PM
For me it consumes even more :). But google bot is always welcomed in my site :)