Results 1 to 11 of 11
  1. #1

    VPS server crashes due to high load - http connections?

    Hello,

    My webserver (VPS) keeps crashing due to high load.

    - Webserver specs (EuroVPS – CT -2800P):
    Guaranteed CPU Speed: 2 Ghz
    Guaranteed RAM: 512 MB

    - There are approximately 100-250 people online at the same time (Whos.Amung.us counter),

    - The actual number of active connections is lower, Munin NetStat never counts more than 100 active connections.

    - Yesterday our site got approximately 12000 visits, 70000 pageviews and people stayed on average 8 min.

    - The pages are all generated html (no php, not dynamic), the only connection to mysql is the advertisement (openx).

    - MySQL doesn’t have much work (queries) with the advertisements.
    At the moment of writing there around 30 active connections, and the CPU load averages are: 0.68 (1 min) 1.47 (5 mins) 9.52 (15 mins)

    We have submitted a support ticket to EuroVPS, and they told us:
    “We confirm that the issue is due to the huge number of http connections to your server which is causing the load.”

    Is it normal that a 2Ghz VPS can’t handle this kind of traffic? Can we do something to fix it? Is the huge number of http connections indeed the cause of the server crash?

    We hope you can provide us with some kind of explanation and/or a solution to this problem.

    Greetings,
    Adrian

  2. #2

  3. #3
    Code:
    Version: 2.5
           uid  resource           held    maxheld    barrier      limit    failcnt
          8610: kmemsize       12989346   13109452   24606720   27340800       2299
                lockedpages           0          0        512        512          0
                privvmpages      133806     136031     230400     256000          0
                shmpages           5878       5878      43008      43008          0
                dummy                 0          0          0          0          0
                numproc             115        115        240        240          0
                physpages         56918      57563          0 2147483647          0
                vmguarpages           0          0     128000 2147483647          0
                oomguarpages      56918      57563      52224 2147483647          0
                numtcpsock           69         70        720        720          0
                numflock             18         19        338        376          0
                numpty                1          1         32         32          0
                numsiginfo            0          0        512        512          0
                tcpsndbuf        296688     301160    2113536    3522560          0
                tcprcvbuf        263512     272592    2113536    3522560          0
                othersockbuf      39360      39360    1373184    3522560          0
                dgramrcvbuf        2800       3500     536576     536576          0
                numothersock         29         29        360        360          0
                dcachesize            0          0    4093977    4548864          0
                numfile            3045       3062      11640      11640          0
                dummy                 0          0          0          0          0
                dummy                 0          0          0          0          0
                dummy                 0          0          0          0          0
                numiptent            14         14        256        256          0
    (Edit: a more readable version here: http : / / upload.vanmaarten.be/2/1/output.jpg )

    kmemsize and privvmpages are often mentioned in our Virtuozzo QoS alerts

    At the moment the server seems a bit stable again though. With currently 180 users online and 60 active connections the load average is at 0.99 (1 min), 0.60 (5 mins) and 0.70 (15 mins), whereas a lesser amount of users and connections caused a crash earlier this morning
    Last edited by ADD3AN; 09-07-2009 at 08:43 AM.

  4. #4
    Join Date
    Jun 2008
    Location
    India
    Posts
    129
    Try to check number of httpd connections at the time of high load

  5. #5
    http: //upload.vanmaarten.be/4/1/connections.jpg

    This is a Munin graphic of our connections. The vertical black lines show where the server crashed.

  6. #6
    I think you should get optimized your apache and mysql. Hope this might solve your problem.
    Support Facility | 24/7 web hosting technical support services
    Technical support | Server management | Data migration

    Technical Articles

  7. #7
    It looks like the Apache server shuts down.

    When I look at the log files, the errors before a crash are always similar to this:

    Code:
    [Sun Sep 06 19:41:44 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:41:54 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:42:04 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:42:14 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:45:12 2009] [error] (12)Cannot allocate memory: apr_pollset_poll: (listen)
    [Sun Sep 06 19:45:13 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:45:14 2009] [error] make_obcallback: could not import mod_python.apache.\n
    Traceback (most recent call last):
      File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 24, in ?
        import pdb
      File "/usr/lib/python2.4/pdb.py", line 9, in ?
        import cmd
      File "/usr/lib/python2.4/cmd.py", line 48, in ?
        import string
    ImportError: No module named string
    [Sun Sep 06 19:45:14 2009] [error] make_obcallback: Python path being used "['/usr/lib/python24.zip', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/local/lib/python2.4/site-packages', '/usr/lib/python2.4/site-packages', '/var/lib/python-support/python2.4']".
    [Sun Sep 06 19:45:14 2009] [error] python_handler: no interpreter callback found.
    [Sun Sep 06 19:45:23 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:45:33 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:45:44 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:45:54 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:49:07 2009] [emerg] (12)Cannot allocate memory: couldn't grab the accept mutex
    [Sun Sep 06 19:49:07 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
    [Sun Sep 06 19:49:17 2009] [alert] Child 6136 returned a Fatal error... Apache is exiting!
    There are also a lot of these errors:
    [Sun Sep 06 18:59:53 2009] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Sun Sep 06 18:59:54 2009] [emerg] (22)Invalid argument: couldn't grab the accept mutex

  8. #8
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    2,955
    Hi there,

    You have two issues based on the above user_beancounters output. The first one is a lack of kernel memory that's allocated to you (controlled by the provider) and a low limit on the number of processes you can launch (also controlled by the provider). Both of which need to be increased so you don't experience those issues anymore.

    Regards
    UNIXy - Fully Managed Servers and Clusters - Established in 2006
    [ cPanel Varnish Nginx Plugin ] - Enhance LiteSpeed and Apache Performance
    www.unixy.net - Los Angeles | Houston | Atlanta | Rotterdam
    Love to help pro bono (time permitting). joe > unixy.net

  9. #9
    Thanks for the suggestions!

    We have tweaked our httpd config a bit and send a support ticket to have both kmemsize and privvmpages increased. Since the changes the server has been running for over a day with a low load average and without a crash, which was impossible before.

  10. #10
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    2,955
    Hi again,

    I would also ask for an increase in numproc.

    Regards
    UNIXy - Fully Managed Servers and Clusters - Established in 2006
    [ cPanel Varnish Nginx Plugin ] - Enhance LiteSpeed and Apache Performance
    www.unixy.net - Los Angeles | Houston | Atlanta | Rotterdam
    Love to help pro bono (time permitting). joe > unixy.net

  11. #11
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    The main thing is to configure apache not to launch too many processes both so you don't exceed your process limit as well as your memory limit. Mysql should also be tuned with a max-connection similarly so you don't exceed your memory limit when apache+mysql usage is combined.

    An easy way to cut back on the number of apache process is to immediately disable keepalives or reconfigure it with a very low timeout, like a few seconds.

    mysqltuner.pl works wonders in helping you figure out how to best tweak mysql both for performance and available memory.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

Similar Threads

  1. server load high, server crashes daily, need help
    By ibuddy in forum Web Hosting
    Replies: 9
    Last Post: 01-03-2006, 02:28 PM
  2. Server load climbs and crashes eventually every few hours
    By GeorgeC in forum Hosting Security and Technology
    Replies: 6
    Last Post: 02-01-2005, 05:55 PM
  3. Linux Stability (2.6, high load) - how many crashes are 'normal'
    By freeflight2 in forum Hosting Security and Technology
    Replies: 4
    Last Post: 09-09-2004, 12:46 PM
  4. Load 99% of the CPU server crashes
    By Hecctor in forum Hosting Security and Technology
    Replies: 10
    Last Post: 07-24-2004, 03:01 AM
  5. Limiting http connections to a server?
    By LMS MIS in forum Programming Discussion
    Replies: 6
    Last Post: 11-05-2003, 11:40 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •