Web Hosting Talk







View Full Version : What is the Google bot doing?!


DevilDog
09-27-2006, 09:42 AM
I give away some web hosting some folks (storage and bw are pretty limited). Once site consistently exceeds it's bandwidth so I decided to figure out why. This shocked me: the google bot, over only about 10 visits, generated about 20,000 hits in only 11 visits! It used over half the bandwidth of the account.

Is there a way to stop this and is it normal?

Jimerson
09-27-2006, 10:15 AM
Look into the robots.txt file. (do a google)


You can tell the bot where to go, or not let it look around at all.

-Jim

Teh_Winnar
09-27-2006, 03:04 PM
Look into the robots.txt file. (do a google)


I find that extremely ironic...

Jimerson
09-27-2006, 03:28 PM
I thought so as well. Thats why I said google and not another search engine.