Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Spiders Causing Excessive Server Load

[NVB]

Today someone brought my server load to more than 22 while attempting to download various forums with a spider program. It took me a while to figure out what was going on and limit the access for his IP address's.

What can I do in the furture to either automatically ban IP addresses that cause excessive server load, or prevent people from spidering the forums (I already have a robots file set up).

[insiderhosting]

well you can use IP chains to block out that ip address, or you could put an .htaccess file in your root directory which will block the spiders from accessing or hammering your site.