Results 1 to 7 of 7
  1. * help: httpd processes stuck with high load

    Hello,

    From time to time I register high loads on my server.
    It is a P4-3Ghz with SATA hard drive and 1GB RAM.
    CentOS 4.3 and apache 2.0.59 and PHP 4.4.4

    The ODD thing is that the load is caused by stuck httpd processes:

    Say in the example attached screenshot I have 3 stuck httpd process running for 39 minutes each!

    Now I have a load of 3+ and it sometimes goes up to 6+ or 7+ with more httpd processes stuck for many more minutes.

    What i do now is manually log in as root; stop and start apache, stuck httpd processes go away. A few hours later, same problem comes back.

    Questions:

    1. What is this? I do not see this with my other servers.

    2. Is this caused by bad reading on the hard disk IO? Hardware?

    3. is this an r57 or c99shell running? Hackers?

    Any help will be appreciated.

    See attached screen shot.
    Attached Thumbnails Attached Thumbnails httpd_stuck_processes_highl.gif  

  2. #2
    Join Date
    Nov 2004
    Location
    Northamptonshire
    Posts
    56
    The CPU usage is in userspace so I am guessing it is a stuck PHP task or similar.
    If it was hardware/IO problems you would see a higher 'wa' percentage.
    I have only seen this happen with one PHP script that had a very bad implementation of AJAX.

    You can look in the error log first. Then if you look in /proc/26807/ using ls -l and its subdirs it may give you a better clue of what is going on, beyond that you may need to break out gdb.
    Andrew Hutchings (A-Wing) - Linux Jedi
    A-Wing Internet Services
    Windows is the path to the darkside...Windows leads to Blue Screen. Blue Screen leads to downtime. Downtime leads to suffering...I sense much Windows in you.

  3. #3
    Setup mod_status and enable ExtendedStatus.

    See mod_status documentation at httpd.apache.org/docs/1.3/mod/mod_status.html

    Then you can point your broser to h__p://machine/server-status and see (among others) table with PID of server process, requested VHost and requested URLs. With this you can find out buggy script or url or something.

    BTW, individual apache processes can be killed independed of each other, so no need to restart whole webserver.

    (Damned forum SW: it is really helps in technical discussion if you cannot use hyperlinks)

  4. #4
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,681
    I would also advise using mod_status or stracing the pid to see if it is stuck in some form of loop.

    Many things can cause a loop however primarily it is addon modules, such as ffmpeg-php does have a tendancy to continue loop writes on certain scripts.

    -Scott
    Server Management - AdminGeekZ.com
    Infrastructure Management, Web Application Performance, mySQL DBA. System Automation.
    WordPress/Magento Performance, Apache to Nginx Conversion, Varnish Implimentation, DDoS Protection, Custom Nginx Modules
    Check our wordpress varnish plugin. Contact us for quote: sales@admingeekz.com

  5. #5
    Uncomment the blow lines and chechk the status of your apache as mentioned above by Scott

    # ExtendedStatus On


    #<Location /server-status>
    # SetHandler server-status
    # Order deny,allow
    # Deny from all
    # Allow from .example.com
    #</Location>


  6. #6
    Not so sure the problem is http but a combination of a kernel and centos.

    if you have kernel slightly before 2.6.9-39.0.2.EL... then it might be the cause f all your misery.

    in termnal type uname -r to see what kernel you have or system details in whm

    if this is the case ask dc to up kernel

    I had the problem on several servers and after a kernel upgrade all came back to normal

    Symptoms on servers were

    many processes for no apparant reason on http (httpd restart brought temp solution)
    usage of swap even when 50 % of ram was free
    high io stats (swap)

    Can't tell for sure but that's my experience, hope this helps you

  7. Wow, thanks for your tips guys.

    I did an ls -l /proc/12345 on the stuck processes and was able to see the cause of incidents.

    The stuck httpd processes are caused by a message board running SimpleMachinesForum 1.1RC3

    The puzzling thing is I have several of these forums running, but only one particular forum is having consistent problems at least 1 stuck httpd process per day.

    I've resorted to fixing the sql database, dumping and dumping back.
    I did a yum update of centos 4.3 and upgraded gcc and the kernel.
    I've re-compiled apache 2.0.59

    Today I still saw 1 httpd process stuck.

    I'll do next the extended server status and learn more and report to you my results.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •