Every now and again, sometimes ten or twenty times a day, the load average on my server will spike dramatically until I restart apache. I've written a script that monitors how many processes apache has and automatically restarts it if there are more than fifty, but that isn't really a suitable way to deal with the problem.
That isn't to say that apache is at fault, although it's definitely possible. When stracing apache processes at any stage I see a lot of EAGAIN notices so I guess it could have something to do with ram, although 512mb should be more than enough for it given it only has a few sites on it and probably only handles ten thousand uniques a day across the board.
Anyway, I am offering $50 to whoever can help me with this problem and give me enough information to fix it.