Results 1 to 7 of 7
-
08-30-2006, 05:09 AM #1Newbie
- Join Date
- Jul 2005
- Posts
- 9
help: httpd processes stuck with high load
Hello,
From time to time I register high loads on my server.
It is a P4-3Ghz with SATA hard drive and 1GB RAM.
CentOS 4.3 and apache 2.0.59 and PHP 4.4.4
The ODD thing is that the load is caused by stuck httpd processes:
Say in the example attached screenshot I have 3 stuck httpd process running for 39 minutes each!
Now I have a load of 3+ and it sometimes goes up to 6+ or 7+ with more httpd processes stuck for many more minutes.
What i do now is manually log in as root; stop and start apache, stuck httpd processes go away. A few hours later, same problem comes back.
Questions:
1. What is this? I do not see this with my other servers.
2. Is this caused by bad reading on the hard disk IO? Hardware?
3. is this an r57 or c99shell running? Hackers?
Any help will be appreciated.
See attached screen shot.
-
08-30-2006, 06:18 AM #2Junior Guru Wannabe
- Join Date
- Nov 2004
- Location
- Northamptonshire
- Posts
- 56
The CPU usage is in userspace so I am guessing it is a stuck PHP task or similar.
If it was hardware/IO problems you would see a higher 'wa' percentage.
I have only seen this happen with one PHP script that had a very bad implementation of AJAX.
You can look in the error log first. Then if you look in /proc/26807/ using ls -l and its subdirs it may give you a better clue of what is going on, beyond that you may need to break out gdb.Andrew Hutchings (A-Wing) - Linux Jedi
A-Wing Internet Services
Windows is the path to the darkside...Windows leads to Blue Screen. Blue Screen leads to downtime. Downtime leads to suffering...I sense much Windows in you.
-
08-30-2006, 11:53 AM #3Newbie
- Join Date
- Jun 2006
- Posts
- 5
Setup mod_status and enable ExtendedStatus.
See mod_status documentation at httpd.apache.org/docs/1.3/mod/mod_status.html
Then you can point your broser to h__p://machine/server-status and see (among others) table with PID of server process, requested VHost and requested URLs. With this you can find out buggy script or url or something.
BTW, individual apache processes can be killed independed of each other, so no need to restart whole webserver.
(Damned forum SW: it is really helps in technical discussion if you cannot use hyperlinks)
-
08-30-2006, 12:05 PM #4Engineer
- Join Date
- Jan 2005
- Location
- Scotland, UK
- Posts
- 2,681
I would also advise using mod_status or stracing the pid to see if it is stuck in some form of loop.
Many things can cause a loop however primarily it is addon modules, such as ffmpeg-php does have a tendancy to continue loop writes on certain scripts.
-ScottServer Management - AdminGeekZ.com
Infrastructure Management, Web Application Performance, mySQL DBA. System Automation.
WordPress/Magento Performance, Apache to Nginx Conversion, Varnish Implimentation, DDoS Protection, Custom Nginx Modules
Check our wordpress varnish plugin. Contact us for quote: sales@admingeekz.com
-
08-31-2006, 06:00 AM #5Disabled
- Join Date
- Aug 2006
- Posts
- 56
Uncomment the blow lines and chechk the status of your apache as mentioned above by Scott
# ExtendedStatus On
#<Location /server-status>
# SetHandler server-status
# Order deny,allow
# Deny from all
# Allow from .example.com
#</Location>
-
08-31-2006, 04:05 PM #6New Member
- Join Date
- Sep 2003
- Posts
- 3
Not so sure the problem is http but a combination of a kernel and centos.
if you have kernel slightly before 2.6.9-39.0.2.EL... then it might be the cause f all your misery.
in termnal type uname -r to see what kernel you have or system details in whm
if this is the case ask dc to up kernel
I had the problem on several servers and after a kernel upgrade all came back to normal
Symptoms on servers were
many processes for no apparant reason on http (httpd restart brought temp solution)
usage of swap even when 50 % of ram was free
high io stats (swap)
Can't tell for sure but that's my experience, hope this helps you
-
09-03-2006, 11:01 PM #7Newbie
- Join Date
- Jul 2005
- Posts
- 9
Wow, thanks for your tips guys.
I did an ls -l /proc/12345 on the stuck processes and was able to see the cause of incidents.
The stuck httpd processes are caused by a message board running SimpleMachinesForum 1.1RC3
The puzzling thing is I have several of these forums running, but only one particular forum is having consistent problems at least 1 stuck httpd process per day.
I've resorted to fixing the sql database, dumping and dumping back.
I did a yum update of centos 4.3 and upgraded gcc and the kernel.
I've re-compiled apache 2.0.59
Today I still saw 1 httpd process stuck.
I'll do next the extended server status and learn more and report to you my results.