For about 15 hrs ago I found out that the I couldnt FTP to my server, I could ping it though. Later I asked for a reboot and then the server worked for about 10 minutes then it was the same. Now I asked for another reboot, SSH'd to the server and ran TOP. This is what I saw:

--- after 1 minute ----

5:05am up 1 min, 1 user, load average: 3.14, 0.97, 0.34
202 processes: 196 sleeping, 6 running, 0 zombie, 0 stopped
CPU states: 94.8% user, 5.1% system, 0.0% nice, 0.0% idle
Mem: 1028456K av, 211416K used, 817040K free, 1288K shrd, 19844K buff
Swap: 1024088K av, 2068K used, 1022020K free 52932K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
981 apache 14 0 41356 40M 6972 R 80.3 4.0 0:29 httpd
1044 apache 9 0 7372 7372 6968 S 1.1 0.7 0:00 httpd
990 apache 9 0 7360 7360 6972 S 0.9 0.7 0:00 httpd
1026 mellow 14 0 1176 1176 840 R 0.9 0.1 0:00 top

--------------------------------------------

--- after15 minutes ----

5:18am up 15 min, 1 user, load average: 74.32, 53.54, 27.72
460 processes: 384 sleeping, 76 running, 0 zombie, 0 stopped
CPU states: 95.2% user, 4.7% system, 0.0% nice, 0.0% idle
Mem: 1028456K av, 1023632K used, 4824K free, 420K shrd, 4472K buff
Swap: 1024088K av, 781348K used, 242740K free 64964K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
1047 apache 20 0 8728 6860 3476 R 1.5 0.6 0:01 httpd
1048 apache 20 0 16452 9496 3500 R 1.5 0.9 0:05 httpd
1098 apache 20 0 33088 7480 3492 R 1.5 0.7 0:21 httpd
1110 apache 20 0 16412 9540 3488 R 1.5 0.9 0:05 httpd
1115 apache 20 0 33100 7524 3480 R 1.5 0.7 0:21 httpd
1128 apache 20 0 16412 9668 3496 R 1.5 0.9 0:05 httpd

---------------------------

It seems like it just eats up all memory (1 GB!!!) and never relases it to new httpds and such? It has been running for several months before in a row without any glitch (with much more traffic than it got now). We havent changed anything to the server. What could be wrong?

Could it be that a harddrive is full or what can make this unpleasant stuff happening?

Thank you,

Michael
Webmaster

The server dies because of the bug in the kernel.
As far as I know (I'm not a *nix specialist) it's a very-very old bug that isn't fixed yet. Server crushes when it eats all memory. In new versions of kernel, it'll start killing all processes including system ones and daemons to free up some memory.

Try to optimize your apache (better search for some manuals - it's to long to describe it here) and run ps -aux right before the server crushes to see the procs. list. May be one of your users launches a lot of perl processes (just for example) and each process will eat 2MB of memory...

Then do something to limit processes running per user (I know it can be done)...

Originally posted by caS
The server dies because of the bug in the kernel.

I seriously doubt this is a kernel bug.

If I were you, I would kill apache immediately after your reboot, then start scanning the logs for funny activity.

the bug is that server halts if there's no free memory.

Well it should not use this much memory. I don't run any scripts and don't got any other users on the server. I usually got pretty many httpds running at once but that should be normal and would not eat 1 gb of memory.

The server don't exactly crash, can still ping it but can't access anything. It has been running fine for months with much more traffic than I got now.

What version of Apache is this? Someone may be scanning/targetting your machine...

httpd -v

If it's anything less than 1.3.26, upgrade to 1.3.26 and see if you have the same problems.

Check the apache status page and logs to see if you have a rogue script.

Hmm, we enabled httpd again and ran TOP. First it comes on HTTPD that eats like 90% CPU, then they divided into 2 or 3, sometimes 4 which alltogether ate like 90%+ cpu. Those are eating memory, we tried to kill ém, but they kept coming back. What could this be?

Do this:

Run Top and hit " c "

thats the command line which will tell you which process are using memory. and try to find it .

Looks like you are getting flooded with hits, but i am not sure :rolleyes:

this is just what I found out, seems to be just like every other httpd there is... :rolleyes:

1851 apache 14 0 35884 35M 6948 R 79.3 3.4 0:21 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DSSL -DSSL.default

Talked to the support, the only thing they think it is, is heavey site usage that cause the high cpu and memory-use. We have been running the same server pushing 10mbit three months without a flaw and now we only push 4mbit so that should not be the issue. We're just running plain html + jpeg/gifs and no scripts.

The CPU-Load should not be 100% and 1 GB of memory should last more than 10 - 15 minutes, right?

Do you have large image files, looks like you are serving extremly large image or movie files. Whats happening is some of your custoemrs are on slow connection and when they try to open these large image files apacxhe stores them till they are fully opened inte custoerms comp. So looks like most of the custoerms you are attracting are on relatively slow connection thus consuming memory and cpu serve those files :D

This is just my theory, I may be wrong .

;)

Are you using SCSI drives, some SCSI drivers need a different number of selectors compiled into the driver module. If its the wrong number you will get a system the continuesly eats memory and gets slower and slower until you need to reboot.