We've been running 2 web servers for our own use for a few years now. We're not sys admins, but have learned a lot in this time! We sometimes have a problem where a server gets high load and takes the server down effectively - we now have a couple of hundred domains on each server - and because we don't know as much about what we are doing than we should we are blind to how to determine what caused it. What would be useful is what other people do to monitor load (we have installed Cacti).
Ideally it would be great if we could pinpoint WHAT is causing the load and also WHO - ie which of the sites/domains on the server are causing it.
What OS are you running (RedHat, CentOS, Suse, FreeBSD, etc...)
Are you running a control panel? (cPanel, DirectAdmin, etc...)
Are you running Apache 1.3, 2.0, 2.2, LiteSpeed, or something else?
Are you running PHP via DSO, FastCGI, suPHP, etc...?
All of these are important questions to help answer your question.
I personally wrote a small daemon that runs and logs the top 50 CPU using applications shown in "ps aux" to a file every 5 minutes for servers not running a system that would log this automatically but a better idea is to catch the load as it's going up - identify it at that point (usually via "top") and then stop it before it crashes.