For the past several days my (Xen-based) VPS has been locking up once or twice a day. Oddly enough I can still ping it, but all websites are down, mail is not delivered, and I can't even SSH in. I end up having to reboot it from the web control panel.
I started collecting stats with MRTG and found that around the time it locks up, my free swap space drops to zero and my load average goes from <1 to around 25. However there are no traffic spikes during these times from what I can tell, and the CPU% remains fairly constant (less than 25%).
So, what else can I do to further troubleshoot this problem? Could it be a problem with my provider's host server? Could the increase in load average be caused by disk thrashing due to high swap usage? I've seen before where the kernel will kill a process that tries to use too much memory but that's not happening in this case. In fact I can't find anything at all in my logs that is suspect.
Any help or insight to this problem would be greatly appreciated!
0.5G RAM + 1G swap (again it's Xen, so that's my own dedicated memory, not "burst"). I've pretty much narrowed it down to Apache, because if I restart httpd then my available swap goes back up and my load average goes back down.
Is there any possible way to determine how much each Apache module is using? I've got several modules I've compiled myself (PHP 5.1.6, Subversion 1.4) plus a couple others I installed from binary (such as mod_python). I've never had problems until very recently so I'm not sure exactly what could be the culprit, and I'd hate to have to disable one for a while.
Of course I could just be trying to run too much; I will probably end up upgrading to the 1G RAM + 2G swap plan if I can't find another culprit.