Results 1 to 5 of 5
  1. #1
    Join Date
    Jul 2006
    Posts
    42

    googlebot...30+ GB bandwidth per month?

    I recently checked my stats page and noticed an anomaly-googlebot is consuming about 30GB of bandwidth on my site per month.

    if I wanted to block googlebot from www.mysite.com/directoryA/fileZ.php, how exactly would I do so? What would robots.txt look like, and where would I put robots.txt, in the directoryA directory?

  2. #2
    Join Date
    Aug 2005
    Location
    UK
    Posts
    654
    If it's just googlebot I'd setup a google site map.. That way you can tell it how often you expect the content on each page or area to be updated. Then you don't waste any bandwidth.

  3. #3

  4. #4
    wow thats a lot of bandwidth, make robot text file and put it in dir as files, i still cant belive it ate that much though amazing

  5. #5
    For me it consumes even more . But google bot is always welcomed in my site

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •