Is it possible to write a script or cron job to monitor the memory status and reboot the server when the memory is being heavily used?

I don't think if it's possible.
but CMIIW.

Reboot the server when the memory is heavily used? Well, a better answer to that might be to increase the RAM, or move the biggest site off to another machine? Although I am far from a server expert, I wouldn't think you'd want the machine rebooting because of this???

so you want your site to be down for a couple minutes when it's getting popular? i thought that's a bad thing...

Chicken,

But it would be better than the unexpected shut down of the server due to the memory allocation error. Don't you think so?

Problem with that is that Linux or Unix tends to allocate alot of memory for holding so while the system may show theres no memory left, you actually have plenty to keep the server running fine... Better solution as Chicken siad would be too add more memory...

12:49am up 10:04, 2 users, load average: 191.50, 161.40, 105.30
435 processes: 420 sleeping, 5 running, 10 zombie, 0 stopped
CPU states: 0.3% user, 93.4% system, 0.0% nice, 6.1% idle
Mem: 257816K av, 254440K used, 3376K free, 46164K shrd, 5240K buff
Swap: 131448K av, 131440K used, 8K free 5936K cached

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
16194 httpd 5 0 8096 300 132 D 0 7.2 0.1 0:17 httpd
16647 httpd 0 0 8012 340 44 D 0 7.2 0.1 0:16 httpd
10712 httpd 0 0 9124 284 12 D 0 6.1 0.1 0:15 httpd
16161 httpd 5 0 8016 200 0 D 0 4.5 0.0 0:11 httpd
19131 httpd 10 0 9200 1888 40 D 0 4.2 0.7 0:17 httpd
14763 httpd 0 0 8896 340 0 S 0 2.1 0.1 0:07 httpd
16324 wmanets 1 0 2124 2124 32 D 0 0.9 0.8 0:03 UltraBoard.c
13911 httpd 1 0 8380 1436 88 D 0 0.8 0.5 0:02 httpd
2205 root 2 0 144 8 0 R 0 0.4 0.0 0:04 sshd
21085 httpd 0 0 9220 1484 0 D 0 0.4 0.5 0:07 httpd
16669 httpd 1 0 0 0 0 Z 0 0.4 0.0 0:01 httpd <defun
5 root 2 0 0 0 0 RW 0 0.3 0.0 0:23 kswapd
19175 httpd 0 0 9172 1260 8 D 0 0.3 0.4 0:07 httpd
22118 admin 12 0 580 536 144 R 0 0.3 0.2 7:58 top
16713 httpd 1 0 8044 348 68 D 0 0.3 0.1 0:00 httpd
13647 httpd 0 0 9112 1832 12 D 0 0.2 0.7 0:02 httpd
14715 httpd 0 0 8260 816 0 S 0 0.2 0.3 0:02 httpd

Before adding more memory, all I can to is rebooting... :(

This is possbile. You could run a CRON script, which would check the memory and if too less, reboot the server. But doing something like this would be silli. Cause the memory will not be maxed out for a long time always. So you could check for the a time period too and then reboot.

By the looks of it, you either have some serious memory leaks, or you just plain need more RAM.

In the meantime, if it's not a memory leak issue, you might be able to band-aid things a bit by setting up some additional swap space.

Originally posted by raylin
load average: 191.50, 161.40, 105.30

Umm... I don't think that rebooting is the solution here. If you've got a load average of over 100, you need a more powerful server (or two... or three...).

Whoa, I didn't even notice the load average.

Actually, a lot of that is because the system is so heavily into swap. Operating out of swap slows things down very badly. An extra bit of RAM would probably help that server tremendously.

Holly cow!

I never saw a load average above 150. 191 is a mother of a load average. What type of server is this?

As Travis said, adding more RAM may improve the perfomance a little bit, but the server sounds like its dieing!

I would suggest you to turn on suexec mode.
So you can see whose process took a lot of CPU resources.

If root/system process took a lot of CPU resources, I would sugget you to add more RAM, upgrade your processor, using SCSI Hard drive, and more reliable mainboard.

My server is RaQ4r with 256Mb. How can I add more swap space?

Originally posted by raylin
My server is RaQ4r with 256Mb. How can I add more swap space?

You don't... at least, not without voiding the warantee on the box.

And it wouldn't help much anyway; increasing the swap might make the box less likely to crash, but with those load averages the box will be so slow as to be unusable anyway.

You should find out who/what is using all the memory and cpu power, but the solution will probably be just to get a more powerful server in the end. (Adding more memory will help somewhat, but if you want to do any more than that you'll have to replace the entire system).