Hello,

I just had a server skyrocket to ~700.0 server load. When it was up that high, there were around 1200 processes (I checked "top"). Mostly all of them were sleeping. I am shocked by the 1200 processes, and I'm thinking this had something to do with the high server load.

Anyway, the machine had a 110 day uptime, and had averaged around 1.0 - 3.0 for all of the 110 days. I rebooted the machine (and added some ram while it was down :) ) . I pasted the results from a top command below.

If you guys could take a look at it and let me know if you see anything out of the ordinary, I'd appreciate it! (The machine is a dual p3 1GHZ with 1GB ram and u/160 SCSI drives).

4:30pm up 45 min, 5 users, load average: 2.35, 2.26, 1.67
268 processes: 257 sleeping, 3 running, 2 zombie, 6 stopped
CPU0 states: 10.1% user, 9.5% system, 0.0% nice, 79.3% idle
CPU1 states: 17.1% user, 7.2% system, 0.0% nice, 75.1% idle
4:39pm up 53 min, 5 users, load average: 1.29, 1.64, 1.64
266 processes: 260 sleeping, 1 running, 0 zombie, 5 stopped
CPU0 states: 2.2% user, 11.2% system, 0.0% nice, 86.0% idle
CPU1 states: 4.3% user, 6.5% system, 0.0% nice, 88.1% idle
Mem: 1027804K av, 1003588K used, 24216K free, 364K shrd, 28112K buff
Swap: 530104K av, 328K used, 529776K free 561204K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
10574 root 9 0 1196 1196 816 S 2.9 0.1 1:01 top
23737 root 14 0 1192 1192 816 R 2.7 0.1 0:16 top
25272 nobody 9 0 10816 10M 9156 S 1.5 1.0 0:00 httpd
24437 nobody 10 0 12216 11M 9076 S 0.9 1.1 0:01 httpd
9130 named 9 0 10300 10M 2228 S 0.5 1.0 0:04 named
18665 nobody 10 0 13100 12M 9064 S 0.5 1.2 0:05 httpd
18608 nobody 9 0 12840 12M 9096 S 0.3 1.2 0:04 httpd
18663 nobody 10 0 11948 11M 9064 S 0.3 1.1 0:02 httpd
20477 nobody 9 0 12904 12M 9068 S 0.3 1.2 0:03 httpd
24867 clockcre 19 19 1488 1488 1384 S N 0.3 0.1 0:00 webalizer
18583 nobody 9 0 13708 13M 9080 S 0.1 1.3 0:07 httpd
18657 nobody 9 0 12132 11M 9084 S 0.1 1.1 0:03 httpd
18692 nobody 9 0 11036 10M 9016 S 0.1 1.0 0:01 httpd
23099 nobody 9 0 12364 12M 9064 S 0.1 1.2 0:01 httpd
24593 nobody 9 0 11444 11M 9164 S 0.1 1.1 0:01 httpd
24860 clockcre 19 19 1488 1488 1384 S N 0.1 0.1 0:00 webalizer
24863 clockcre 19 19 1488 1488 1384 S N 0.1 0.1 0:00 webalizer
25209 nobody 9 0 9284 9284 8992 S 0.1 0.9 0:00 httpd
25505 nobody 9 0 9264 9264 8996 S 0.1 0.9 0:00 httpd
25572 root 11 0 2396 2396 1960 S 0.1 0.2 0:00 sendmail
1 root 4 0 524 524 456 S 0.0 0.0 0:04 init
2 root 8 0 0 0 0 SW 0.0 0.0 0:00 keventd
3 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU0
4 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU1
5 root 9 0 0 0 0 SW 0.0 0.0 0:00 kswapd
6 root 9 0 0 0 0 SW 0.0 0.0 0:00 kreclaimd
7 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
8 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated
9 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
10 root 9 0 0 0 0 SW 0.0 0.0 0:00 phpd_notify

It would help if you could provide the same information when the server load was at 700.

If it happens again.

Yes, I know. I was just too busy trying to fix the problem. :D Anyway, I'll be sure to get a copy of it if it shoots up there again.

The main thing I was wondering about is if 250 processes was a bit high. (ie.. this line: "268 processes: 257 sleeping, 3 running, 2 zombie, 6 stopped" )All the other posts I have seen on here have had like 50 processes going.

I suppose if their is nothing jumping out at you for the 'top' above, then that is a good thing. :stickout

Thanks!

Originally posted by mpope
Yes, I know. I was just too busy trying to fix the problem. :D Anyway, I'll be sure to get a copy of it if it shoots up there again.

The main thing I was wondering about is if 250 processes was a bit high. (ie.. this line: "268 processes: 257 sleeping, 3 running, 2 zombie, 6 stopped" )All the other posts I have seen on here have had like 50 processes going.

I suppose if their is nothing jumping out at you for the 'top' above, then that is a good thing. :stickout

Thanks!

The number of processes will depend on the number of hits per second you are getting. Was the server responsive on ssh???

webalizer takes some cpu to calculate the stats.

Well your top didn't show 266 processes. :)

Anyway, if 260 of them are sleeping, and you have plenty of memory, it shouldn't be a problem. However, if most of those are http children, please keep in mind that you don't need to have that many spare servers. Apache is pretty super fast at spawning extra ones if it needs them. If your minspareservers setting is 256, it probably doesn't need to be that high.

I would ask a question here; When the load reported at ~700, was it deathly slow? I ask, because I can't imagane *anything* would respond, at all, if it was anywhere near that. It's hard to imagine how you ran top. If the server was responsive, even if it was slow, it could be a bug. I've seen a bug that will basically just keep adding up processes, even if they've died off, but it doesn't affect the server load. Of course, if you had a high load in addition, it would still be as slow as that high load made it, but it's just hard to imagine it being around 700.

Yes, it was deathly slow. However I was able to SSH into it and ran top. It did take a while to load up, but I did not want to reboot untill I saw the load. However, the server was responding quickly to static HTML requests. (none of the mysql or cgi were working though.)

bitserve - actually that is something I have been looking at raising lately. The minspareservers setting is currently at 10 and maxspare servers is at 30. It was at 5 / 10 when the 700 server load happened...

I would put money on apache not shutting down its processes correctly and you started to hit swap which would make it deathly slow...and have a high load.
But it couldbe other things too....this is just my guess ;)

Could of been a run away process that was taking up all the memory, i have had that happen on my WHM servers a few times, it causes a big back log of process and the load to sky rocket.

Yes, I had originally thought it was a run away process, but nothing seemed "out of the ordinary" in the top.

Anyways, it's been a couple of days and it seems a reboot did the trick! =)

I've found on my server is people dl files from httpd and not FTP it can cause a high server load, what sort of site is it?

Originally posted by mpope
bitserve - actually that is something I have been looking at raising lately. The minspareservers setting is currently at 10 and maxspare servers is at 30. It was at 5 / 10 when the 700 server load happened...

Haha. I was assuming that the majority of your 260 processes were http children. What are they?

If this is a web server, you probably do want to increase your minspareservers and maxspareservers.

There are quite a few threads on here discussing those settings. You could do a search if you want more info.

If you were getting a lot of traffic to your web site and you only have 30 max servers running, a lot of the connections are going to be waiting, and this can increase your server load. There are threads on this too.

I don't know what the heck they are. :)

I have looked through a couple of threads on min/maxspareservers, and most likely will be increasing them. The one thing I read in one of them is someone said you have to recompile apache for the change to take effect. This isn't true is it? I figured I could just change the value in httpd.conf and restart apache and it should be good to go.

Originally posted by mpope
The one thing I read in one of them is someone said you have to recompile apache for the change to take effect. This isn't true is it? I figured I could just change the value in httpd.conf and restart apache and it should be good to go.

You generally don't need the maxspareservers set over say 200. The directive that you generally can't set too high without recompiling is the maxclients (http://httpd.apache.org/docs/mod/core.html#maxclients) setting.

You may not need to recompile even to set this up high. It may have already been compiled for a higher setting. You haven't even said what OS you're using. :)