Web Hosting Talk







View Full Version : Spiders Causing Excessive Server Load


NVB
03-23-2002, 04:09 PM
Today someone brought my server load to more than 22 while attempting to download various forums with a spider program. It took me a while to figure out what was going on and limit the access for his IP address's.

What can I do in the furture to either automatically ban IP addresses that cause excessive server load, or prevent people from spidering the forums (I already have a robots file set up).

insiderhosting
03-23-2002, 04:49 PM
well you can use IP chains to block out that ip address, or you could put an .htaccess file in your root directory which will block the spiders from accessing or hammering your site.