Results 1 to 13 of 13
  1. #1
    Join Date
    Oct 2005
    Posts
    435

    Server spikes randomly

    Everything was running ok until last week.

    I haven't really done anything new other then added another forum on the server (Invision).

    Basically the problem is the random spikes I get. The main forum that is running on the server haven't really increased in terms of sim connections. Its usually around 180 - 220 during peak times.

    On TOP there is usually around 250 tasks. When this thing started happening it would spikes to 500+ tasks which totally kills the server. Its pretty random also. Sometimes it'll go fine for a few hours and then suddenly climbed slowly up and then the server dies. For now its going to crash whenever I let apache run for an hour or so.

    The only way I can restore is to restart apache. It has come to the point where I've had to put in restart apache in the cronjob list. Restart it every few minutes since I'm not always here watching over it. This is obviously a shortterm solution to keep the forum up while I troubleshoot this.

    Here's a screenshot when it was just about dead

    http://img392.imageshack.us/my.php?image=serverwf0.jpg

    I checked out the logs and didn't see anything.

    I was either told it needed more ram or that it was hacked.

    The JRE is the flashchat I'm running and the trans_linux is a bot for the shoutcast.

    Since I haven't been doing anything new.. I don't think its the resources of the server thats too little.

    The server i have is

    Dual Opteron 252
    2 gigs of ram
    1 146GB 15,000RPM scsi drive

    Any idea how I can resolve it? Is Apache using more memory then its suppose to?


    Thanks for reading!

  2. #2
    Check the server status to see exactly what is running.

    Make sure you have the following in httpd.conf
    <Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
    </Location>
    ExtendedStatus On

    Then type
    wget 127.0.0.1/server-status
    lynx server-status

    You will probably see its 1 script or 1 domain causing the overload.

  3. #3
    Join Date
    Oct 2005
    Posts
    435
    Hey oscar

    I tried what you mentioned but I get access denied. I've turned off apf but I still get the same error.

    Also I've managed to solve the problem by turn off the KeepAlive option to Off in httpd.conf

    After that I don't have any problems anymore but yesterday the server still got overloaded twice. So thats still a mystery to me. There's wasn't anymore people in those perticular time either.

    Thanks.

  4. #4
    Did you run the wget command as root?

  5. #5
    Join Date
    Oct 2005
    Posts
    435
    yes I ran it as root.

  6. #6
    Join Date
    Jul 2004
    Location
    Texas
    Posts
    687
    upgrade your ram. you are almost 2gb into swap, that is killing your hd, which is causing i/o to suffer badly.

  7. #7
    Join Date
    Oct 2005
    Posts
    435
    Yea I see that Tailseh

    but I feel that it is something else.

    Would a system like I mentioned above need more then 2 gigs of ram to serve 200+ sim connections (and of course running shoutcast and a small java server flashchat).
    Last edited by Darvil; 01-18-2007 at 12:00 AM.

  8. #8
    Join Date
    Apr 2003
    Location
    San Jose, CA.
    Posts
    1,622
    Why do you feel it's something else?

    Your apache processes alone are using anywhere from 4 to 60MB of resident memory...

    Guestimating an average of 10Mb... 200*10Mb resident memory and you're swapping (not counting all the other proceses) = system craters.

  9. #9
    Join Date
    Oct 2005
    Posts
    435
    Its because the server has been smooth running fine for 2 months with no issues until last week (and we haven't had any big growth or I haven't installed anything big)

    And if you can see in the tasks in the screenshot.. the tasks is almost up to 600.

    That has never happened so I'm thinking its something else.

  10. #10
    Join Date
    Apr 2003
    Location
    San Jose, CA.
    Posts
    1,622
    I still don't get why you think it's something else...

    Simple math and looking at your top output should tell you the system is swapping itself to death... ie. it's ran out of physical memory... all the swapping is causing your IO wait to skyrocket... causing all your processes to run sluggish... it's a downward spiral once your system starts actively swapping.

    Start a:
    sar -rq 1 999999 > watchmecrash.txt&

    then look at the output of the file once the system starts slowing down...

  11. #11
    Join Date
    Oct 2005
    Posts
    435
    Hey Light. I was looking at this http://www.webhostingtalk.com/showthread.php?t=469677

    I did some mysql stuff and its looking better

    Right now the server is as max at its usually is. 200 forum users, 15 chatters and about 20 streaming from shoutcast

    http://img107.imageshack.us/my.php?image=server2bl9.jpg

    This is whats suppose to be a normal amount of tasks from the pic (around 160+)

    I think there were extra mysql processes taking up the tasks. So far its moving as smooth as butter but we'll see.

    I've been recommend to get an extra gig of ram but I kind of want to stretch it out as much as I can with this. The load is a little stressful but it is peak time after all.

    But if it all goes to hell again then we'll see.

    BTW I don't have the "sar" command. (http://www.computerhope.com/unix/usar.htm)

    I have CentOS installed.

    Thanks for your comments and would still like feedbacks.
    Last edited by Darvil; 01-18-2007 at 02:27 AM.

  12. #12
    Join Date
    Apr 2003
    Location
    San Jose, CA.
    Posts
    1,622
    "BTW I don't have the "sar" command. "

    yum install sysstat

    sar and iostat are very helpful tools in understanding what your system is doing.

  13. #13
    Join Date
    Apr 2005
    Posts
    535
    i would also suggest getting at least another gig of ram, preferably 2 gig looking at that top screen, just to be on the safe side

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •