you'll have the dcpumon logs to check the load issues, /var/log/dcpumon/ and also the /var/log/messages file what might have cropped up then
Kevin Cheri : Senior Server Administrator / Freelancer : 9+ years Exp, reach me out for any help Server Optimization Expert / Mysql Guru / Migration Specialist
Skype : lynxmaestro
Gmail : [email protected]
Better put a script that runs every minute and grab necessary data ( like Apache request pool, connection list to port 80, top result, etc ) when server load crosses the threshold. If you want suggestions for already happened issue, check /var/log/dcpumon, /var/log/messages , /usr/local/apache/logs_error_log, and the result of /usr/local/cpanel/bin/dcpumonview command.
Dcpumon logs will get overwritten so you won't be getting much data from there unless you are checking them realtime. If you have lfd running on the server, you can track the excessive process run from /var/log/lfd.log . Also you can get load averages from sar output which reads data from these dcpumon logs. I would suggest to write a script to track the load and send you a mail when the load exceeds a threshold value set. This script should be providing output of top process, mysql queries, memory usage, netstat output etc.
most recent load-spikes I've noticed have been due to proftpd brute force attacks - we have fail2ban running frequently, but over the weekend I noticed one IP range hammering a primary shared server - the load average spiked to 84.5 - where it normally sites < 1.
I firewalled out the chinese IP range and server load reduced within a couple of minutes. This is an ongoing thing, and honestly, I expect the data-centers to start handling some of this load, reacting to logfiles is far too reactive for us - and it will never end.
In order to determine if you have similar issues - login as root or other super-user - tail -f /var/log/secure and tail -f /var/log/messages for a couple of minutes.
Once you find a likely IP - check if they should be firewalled or have been firewalled before. I have a small shell script I put togehter to check my firewall setup and the fail2ban logs: