Results 1 to 15 of 15
  1. #1
    Join Date
    Jul 2003
    Location
    London, UK
    Posts
    116

    * Googlebot Hammering My Website!

    Googlebot has been absolutely ripping through my bandwidth. This has been on going for many months now, but each month the damage has gotten worse and worse. I have posted about this problem here but with no luck:
    http://www.webmastertrader.com/showthread.php?t=389

    To demonstrate the problem, I have decided to show you all my server stats for one of my websites. I have no choice but to ban googlebot from my site and follow this action with a letter to google!
    1) Aug: http://www.webmastertrader.com/attac...0&d=1159816704
    2) Sep: http://www.webmastertrader.com/attac...1&d=1159816714
    3) Oct: http://www.webmastertrader.com/attac...2&d=1159816723

    We're only a day and a half into this month and Google's robots have already consumed over 1gig of bandwidth!

    Any help would be appreciated!
    BioALIEN
    █ Secure your name before someone else does: The Million Dollar Advert.com
    █ Get your permanent keyword listing and get better Search Engine ranking.

  2. #2
    Join Date
    Mar 2005
    Location
    Maine, USA
    Posts
    302
    Googlebots are suppost to be robots.txt compliant, so if you put a robots.txt file in the root directory of each domain, they should read it and comply with what it says there.
    http://www.google.com/support/webmas...y?answer=40364

    You could also block the ip addresses of the bots in your firewall. APF makes it easy to block IP addresses.

    Btw, those links there require a login, as I do not have one there. And I won't, so you might want to repost those messages here or attach them.

  3. #3
    Join Date
    Oct 2002
    Location
    Canada
    Posts
    3,100
    If this is regarding the site in your sig, in robots.txt you could put something like this:

    User-agent: googlebot
    Disallow: *album_pic.php*
    Disallow: *album_thumbnail.php*

    That way google will still spider your site but it will not download any images

  4. #4
    Join Date
    Jul 2003
    Location
    London, UK
    Posts
    116
    hawk82, I have attached the images to this forum. They should work now!

    sasha, thanks for the tip I'll give it a try.
    Attached Thumbnails Attached Thumbnails august.gif   september.gif   october.gif  
    BioALIEN
    █ Secure your name before someone else does: The Million Dollar Advert.com
    █ Get your permanent keyword listing and get better Search Engine ranking.

  5. #5
    Join Date
    Feb 2003
    Posts
    543
    Sorry I havent read the relating threads but have you confirmed that it actually is Google. I know some people are imitating them recently.
    Off Topic Web Forum - A forum for talking about anything!!
    N.Z. Webmaster Community - Are you from New Zealand? Well signup to our forum!!!!

  6. #6
    Join Date
    Jul 2003
    Location
    London, UK
    Posts
    116
    PhilG, as you can see from the screenshots provided by AwStats, they are identified by Googlebot. Looking at the live sessions, they do seem to be coming from legit google IPs.

    I have no enforced a robots.txt but the problem is still present.

    Googlebot
    98695+1
    3.25 GB
    Bandwidth consumed has more than doubled in the period of 24 hours!
    BioALIEN
    █ Secure your name before someone else does: The Million Dollar Advert.com
    █ Get your permanent keyword listing and get better Search Engine ranking.

  7. #7
    lol.. Wow..

    Sorry man I have no solution but that is insane,.,

  8. #8
    Join Date
    Feb 2003
    Posts
    543
    I would contact Google directly for this.

    http://www.google.com/support/bin/request.py

    It’s by all means unusual and even if you have no robots Google shouldn’t thrash your site like this.
    Off Topic Web Forum - A forum for talking about anything!!
    N.Z. Webmaster Community - Are you from New Zealand? Well signup to our forum!!!!

  9. #9
    I've faced this problem in the past. Google crawled my site at the rate of more than 300K hits a day and almost pulled the whole server down. I contacted Google and request them to slow down crawling rate on my site. They responded promptly, asked me to upload a forgoogle.txt file to confirm my ownership of the site and then Googlebot became gentle again.

  10. #10
    Join Date
    Jul 2003
    Location
    London, UK
    Posts
    116
    PhilG, I have exercised every option available to me but I'll contact Google Support as Kamejoko pointed out.

    Thanks for the feedback guys, if theres anything else I can do in the meantime please dont hesitate to voice it.

    Just a sidenote, this problem started October 2005 and one year on, it has cost me a fortune to keep the website going. I have tried in the past robots.txt and blocking a few Googlebot IPs but with no use.
    Last edited by BioALIEN; 10-04-2006 at 06:25 AM.
    BioALIEN
    █ Secure your name before someone else does: The Million Dollar Advert.com
    █ Get your permanent keyword listing and get better Search Engine ranking.

  11. #11
    Join Date
    Aug 2006
    Location
    Western Europe
    Posts
    217
    Hello,

    In addition of a robots.txt file, I would advise you to use a .htaccess file to forbid google to come and mess with you.

    Put this :

    <Directory /picturesdirectory>
    Order Deny, Allow
    Deny from 66.249.64.0/19
    Deny from *.google.com
    Deny from *.googlebot.com
    Allow from all
    </Directory>
    Note: I am not sure about if google does own other IP blocks.
    Last edited by Dualism; 10-04-2006 at 06:58 AM.

  12. #12
    Join Date
    Jul 2003
    Location
    London, UK
    Posts
    116
    Dualism, Google does own a few other IP blocks which I verified and it lead back to them.
    BioALIEN
    █ Secure your name before someone else does: The Million Dollar Advert.com
    █ Get your permanent keyword listing and get better Search Engine ranking.

  13. #13
    Join Date
    Dec 2005
    Posts
    82
    Also I think you can bloack any hosts that contain "google" by mod rewrites:

    RewriteEngine On
    RewriteCond %{REMOTE_HOST} ^.*google.*$
    RewriteRule ^(.*) http://www.redict_page.com/ [R=301]

  14. #14
    Join Date
    Aug 2006
    Location
    Western Europe
    Posts
    217
    Mate, checkout your PM's, I have sent you all their IPs.

  15. #15
    Join Date
    Jul 2003
    Location
    London, UK
    Posts
    116
    Thanks Dualism and ngsupb for the tips. I am currently waiting on the Google support to assist before trying your approach.
    BioALIEN
    █ Secure your name before someone else does: The Million Dollar Advert.com
    █ Get your permanent keyword listing and get better Search Engine ranking.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •