Server performance increases 5x for 6 months then decreases, ideas welcome
Hope someone can throw up an idea for me on this one.
Up until March of this year scripts such as rootkithunter took 250 seconds to run and another perl script of mine can take 200 - 250 seconds to run these are just some amongst many and has been like that since the server went up two years ago, I figured it was normal.
Then one day in March this year rootkithunter takes 45 seconds to run, my other script completes in a little over 30 seconds and all other scripts are considerably faster. Server usage is the same. I run the latest stable version of cpanel on a no frills RedHat 9 box and looked at what may have been updated that night and can only find that glibc and nscd were updated. I investigated for some time but got nowhere and decided a 5x performance increase whilst unplanned wasn`t the worst that could happen :-)
Then the other day I decide to get my DC to update the kernel, after this upgrade and a reboot the machine is back to its old slow ways of pre March :-( I figured it was the new kernel 2.4.32 and got the DC to reboot into the previous kernel 2.4.29 but no joy there, I then rebooted back into the new kernel. I don`t think the machine was rebooted between the surprise performance increase and its subsequent revert back to slowness.
If anyone has any tests they think may throw some light on this for me that would be great, bearing in mind the unusual circumstances of its occurence.
P.S. If anyone out there is running rootkithunter on a hyperthreading P4 3Gb with 1Gb RAM or there abouts with RedHat 9 can you post the time taken for the script to run on your server, thanks. Mine takes on average 300 seconds slow (now) and 40 seconds fast (before).