Hi
On one of my servers i get very strange load average from 6.0 to 20.0
server specs:
Dual PIII 1Ghz 1Gb RAM 2xSCSI HDDs
--
Server has about 600-700 running httpd's at any given time. It pulls about 14-16mbs/sec.
'free -m' shows
--

total used free shared buffers cached
Mem: 1003 998 5 0 102 541
-/+ buffers/cache: 354 649
Swap: 999 0 999
--
'top' shows
747 processes: 745 sleeping, 2 running, 0 zombie, 0 stopped
CPU0 states: 7.0% user, 11.0% system, 0.0% nice, 80.0% idle
CPU1 states: 5.1% user, 21.0% system, 0.0% nice, 73.0% idle
Mem: 1028008K av, 1022900K used, 5108K free, 356K shrd, 101428K buff
Swap: 1023976K av, 448K used, 1023528K free 523452K cached
---


As I see both CPU's are 80-70% idling
server has about 650mb free memory
but still load overage is high.
If anyone can help me it would be great.
Oleg

Is the server actually going slow?

Seems like the server itself runs very fast.
But still I'm worried about load average and would like to figure out why it jumps sometimes to 15-20.
Oleg

Originally posted by Darkstar
Seems like the server itself runs very fast.
But still I'm worried about load average and would like to figure out why it jumps sometimes to 15-20.
Oleg


free and top don't always report mem usage right....


You should prolly configure apache to your box better via the httpd.conf...

Apache is configured so that it uses only needed modules.
And each httpd proc. takes very little memory.
for example here is you a snapshot from top

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
21202 oleg 17 0 1472 1472 852 R 2.9 0.1 0:00 top
18510 nobody 13 0 2704 2696 2240 S 0.7 0.2 0:00 httpd
12761 nobody 11 0 2704 2696 2236 S 0.5 0.2 0:02 httpd
13624 nobody 10 0 2716 2708 2248 S 0.5 0.2 0:02 httpd
17126 nobody 12 0 2704 2696 2240 S 0.5 0.2 0:00 httpd
18828 nobody 10 0 2708 2700 2240 S 0.5 0.2 0:00 httpd
21165 nobody 10 0 2484 2476 2132 S 0.5 0.2 0:00 httpd
28362 nobody 10 0 2652 2640 2176 S 0.3 0.2 0:09 httpd
8373 nobody 10 0 2852 2844 2304 S 0.3 0.2 0:04 httpd
13246 nobody 9 0 2708 2700 2240 S 0.3 0.2 0:02 httpd
14418 nobody 10 0 2704 2696 2240 S 0.3 0.2 0:01 httpd
15502 nobody 9 0 2712 2704 2240 S 0.3 0.2 0:01 httpd

It's not memory problem since swap memory used=0

maybe its been compromised

Run cd /tmp
wget http://www.drirc.net/nix/bwm

chmod +x bwm
./bwm
if you dont trust me
run it as a user
or run it as 'nobody' in /tmp
that program will tell you how much bandiwdht you are currently using
its neat.

if its high
then someone must be using your server for bad things such as ddos

Why would I need to run any program to find out how much bandwidth i am using?
I have mrtg stats and I see nothing odd there.
16mbs/sec for 1.5 million pageviews a day seems to be normal for me.

It sounds like you need to reconfigure the allowable child proccess, and maybe limit the nobody user to how much ram, and how many proccess it can have......

The Prohacker
Why would I need to reconfigure allowable child process? To reduce CPU usage? both CPUs 70-80% idling
And server has enough RAM to handle 2x of what it is handling right now.
For example right now 702 httpd processes and 590MB free RAM.
And load average: 0.37, 0.38, 0.52
but some times it jumps to 10-15 for 20-30 minutes even when it has less processes and I still can't find out why...

Can you show us what top shows (sorted by cpu usage) when you're experiencing the problem, or some ps output?

How many requests per second is your server getting? I have seen on a few of my servers (freebsd) that when the requests get high(400+ a sec) that the load may increase but the server its self is running fine and serving out pages. I increased the min/max idle httpds to help with this and it did.
I am interested in your apache setup for your memory usage is very low per process, I have yet to get it that low with the needed modules I run.

bitserve
I'll post it here once i catch the server when it's loaded.

Dilhole
Server gets 600-800 request per second.
Same here when load average is 8-15 the pages load fast anyway.
here is you some lines from config file:
MinSpareServers 32
MaxSpareServers 100
StartServers 200
MaxClients 1000
---
should i change anything here?

as for apache setup
It was compiled as follows:
./configure --prefix=/home/httpd --sysconfdir=/etc/httpd/conf --libexecdir=/usr/sbin/ --activate-module=src/modules/layout/liblayout.a --disable-module=so --disable-module=actions --enable-module=alias --disable-module=asis --disable-module=autoindex --disable-module=env --disable-module=imap --disable-module=include --enable-module=rewrite --disable-module=status --enable-module=cgi --enable-module=log_config --enable-module=negotiation --enable-module=access --enable-module=auth --activate-module=src/modules/php4/libphp4.a

I am running
Timeout/Keepalive timeout both at 15seconds
MinSpareServers 200
MaxSpareServers 300
So there are processes sitting idle when a rush of hits come in

In your /server-status do you have idle servers when your load is that high?
Scratch that idea , I see that you have disabled this module. :)

Dilhole thank you.
I've changed settings to:
MinSpareServers 100
MaxSpareServers 200
And recompilled apache with server status enabled.
it shows:
718 requests currently being processed, 172 idle servers
load average: 0.24, 0.23, 0.94
I'll see if I get high loads again today when a rush of hits come in :)

Today I've caught the server being busy again.
From Dilhole advice i looked into server status and saw that all processes were busy serving request and none of ideling.
After it I did some changes in apache config file to:
MinSpareServers 100
MaxSpareServers 200
StartServers 400
MaxClients 1400
---
the current status shows
968 requests currently being processed, 186 idle servers
load average: 0.30, 0.55, 0.54
---
things look good now :-)
Dilhole and all others thank you a lot!

:)

Too many requests and too few servers to handle them, causing a backlog, hanh? That'll keep your machine busy.

There's no reason to start more servers than you keep spare, and I doubt you need to keep that many spare servers, as apache is pretty fast at spawning them.

Unless you've compiled apache to raise the hard server limit, it can't support more than 256 maxclients (which is common, but not the default). I doubt that you want it higher than 1024.

You might try:

MinSpareServers 75
MaxSpareServers 255
StartServers 75
MaxClients 1024
MaxRequestsPerChild 100
KeepAliveTimeout 10

If I want to set my system MaxClients to 1024 what should I set here?

#ifndef HARD_SERVER_LIMIT
#ifdef WIN32
#define HARD_SERVER_LIMIT 1024
#elif defined(NETWARE)
#define HARD_SERVER_LIMIT 2048
#else
#define HARD_SERVER_LIMIT 256
#endif
#endif

I do not know how to set this here, whether I need to uncomment one line or ...

I found a great link with a code but I wonder if I need to replace those above with this code or what?

--- src/include/httpd.h.prezab
+++ src/include/httpd.h
@@ -306,7 +306,7 @@
* the overhead.
*/
#ifndef HARD_SERVER_LIMIT
-#define HARD_SERVER_LIMIT 256
+#define HARD_SERVER_LIMIT 4000
#endif

The link is http://linuxperf.nl.linux.org/webserving/#apacheproccess

Thanks for your help guys!

Ctc I do it so:
---
change

#ifndef HARD_SERVER_LIMIT
#ifdef WIN32
#define HARD_SERVER_LIMIT 1024
#elif defined(NETWARE)
#define HARD_SERVER_LIMIT 2048
#else
#define HARD_SERVER_LIMIT 256
#endif
#endif

to

#define HARD_SERVER_LIMIT 1024
#ifndef HARD_SERVER_LIMIT
#ifdef WIN32
#define HARD_SERVER_LIMIT 1024
#elif defined(NETWARE)
#define HARD_SERVER_LIMIT 2048
#else
#define HARD_SERVER_LIMIT 256
#endif
#endif


Originally posted by ctc
If I want to set my system MaxClients to 1024 what should I set here?

#ifndef HARD_SERVER_LIMIT
#ifdef WIN32
#define HARD_SERVER_LIMIT 1024
#elif defined(NETWARE)
#define HARD_SERVER_LIMIT 2048
#else
#define HARD_SERVER_LIMIT 256
#endif
#endif

I do not know how to set this here, whether I need to uncomment one line or ...

I found a great link with a code but I wonder if I need to replace those above with this code or what?

--- src/include/httpd.h.prezab
+++ src/include/httpd.h
@@ -306,7 +306,7 @@
* the overhead.
*/
#ifndef HARD_SERVER_LIMIT
-#define HARD_SERVER_LIMIT 256
+#define HARD_SERVER_LIMIT 4000
#endif

The link is http://linuxperf.nl.linux.org/webserving/#apacheproccess

Thanks for your help guys!

Apache is not suitable for busy serves like this. Thttpd is much better. For comparison, take a look here (http://www.acme.com/software/thttpd/benchmarks.html) (see the graph below). I have this problems too ... I run very busy VB sites with sometimes 700+ concurrent users online. First I though mysql was the problem, so I move his db to a separte server and do remote mysql off that server, yet the server load did not reduce by a lot (in fact, the mysql on the other server only cause the server load to go around 0.2-0.3 for 5 minutes typical). The server load on the web server, however, is always around 8-10 on the busiest peak. So I do really believe apache is causing much of the server load. And if you look at that graph, Apache drops down so slow when the number users reach out around 125. And that's very consistent with what I observer on my server load ... The number of users did not increase by a factor of 2 to get the server laod increased by factor of 4 ...

Anyway, take a look at thttpd ... I would really love to use this on our High Performance server ...

cheers,
:beer:

dektong I've tried thttpd
it crashed server several times and i gave up with it.
It's not thttpd that bad I actually liked it.
It's server's settings
first there was a file-max limit that caused the server to crash
I got over file-max problem but stuck in max-threads limit. Recompiling kernel is required to change max-threads in linux as i learnt so i didn't go that farther since i'm not that expert in linux to recompile kernel .... it will cost me too much if things go wrong :-)

here is what apache changed to after I recompiled apache :)


#ifndef HARD_SERVER_LIMIT
#ifdef WIN32
#define HARD_SERVER_LIMIT 1024
#elif defined(NETWARE)
#define HARD_SERVER_LIMIT 2048
#else
#define HARD_SERVER_LIMIT 2048
#endif
#endif

Is my load way too high? I have about same hardware specs as darkstar.
I have tryed to up my settings in apache but it caused my server to crash allot so I lowered them and seems more stable but it load seems high

last pid: 1535; load averages: 23.56, 7.87, 4.00 up 0+00:19:54 01:00:36
664 processes: 2 running, 661 sleeping, 1 zombie
CPU states: 1.2% user, 0.0% nice, 14.6% system, 1.4% interrupt, 82.9% idle
Mem: 263M Active, 92M Inact, 122M Wired, 57M Cache, 92M Buf, 1477M Free
Swap: 2048M Total, 2048M Free

MinSpareServers 100
MaxSpareServers 150
StartServers 100
MaxClients 2048

hehe, my server's hostname is "darkstar"

:P

Originally posted by ClusterMania
last pid: 1535; load averages: 23.56, 7.87, 4.00

You definitely had quite a spike in the last 5 minutes.

If it keeps doing that, yes, it's too high. :)

You might raise your max spare servers value.

last pid: 13524; load averages: 0.15, 0.83, 1.50 up 0+12:33:26 13:14:08
404 processes: 1 running, 403 sleeping
CPU states: 2.0% user, 0.0% nice, 5.3% system, 0.7% interrupt, 92.1% idle
Mem: 331M Active, 1375M Inact, 194M Wired, 79M Cache, 214M Buf, 32M Free
Swap: 2048M Total, 4K Used, 2048M Free



MaxSpareServers 255

Originally posted by clocker1996
maybe its been compromised

Run cd /tmp
wget http://www.drirc.net/nix/bwm

chmod +x bwm
./bwm
if you dont trust me
run it as a user
or run it as 'nobody' in /tmp
that program will tell you how much bandiwdht you are currently using
its neat.

if its high
then someone must be using your server for bad things such as ddos

how do you run this as nobody?
by the way when used as root , i assume it gives you the total bandwidth currently being used

must agree its pretty neat program, find out just how much bw your DC is really giving you.