Today i went and ran top. I noticed that the majority of the commands were httpd, but it said <defunct> beside it. what does this mean? is it bad? what can I do about it.

thanks

upon further inspection I also notice that almost all connections are TIME_WAIT, yet my logs dont show an outlandish amount of traffic.

I'm confised

Defunct processes (actually 'zombies' , note the 'Z' under STATus , are just programs that have quit, but that the system hasn't removed from memory yet. It takes a little while for the kernel to clean up after a program is done, during that short interval they are 'defunct'.

The only reason you see it with httpd so much is it's very active and starts/quits it's sub processes fairly often, but if you watch closely, you will probably see other things doing it occasionally.

Zombies that hang around and won't go away are another matter however; If the same defunct processes stay there for a long period of time post a copy of tour top display.

11:50pm up 2 days, 1:14, 1 user, load average: 0.73, 1.01, 1.11
173 processes: 162 sleeping, 2 running, 9 zombie, 0 stopped
CPU states: 5.7% user, 30.2% system, 0.0% nice, 64.0% idle
Mem: 506236K av, 494956K used, 11280K free, 0K shrd, 15180K buff
Swap: 1048280K av, 5080K used, 1043200K free 390532K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
915 root 16 0 6192 6192 6020 S 3.7 1.2 0:08 httpd
8761 apache 15 0 6628 6628 6260 S 0.9 1.3 0:00 httpd
7696 root 16 0 1116 1116 832 R 0.5 0.2 0:00 top
8511 apache 15 0 0 0 0 Z 0.3 0.0 0:00 httpd <defunct>
8853 apache 15 0 6432 6432 6224 S 0.3 1.2 0:00 httpd
8880 apache 15 0 6340 6340 6128 S 0.1 1.2 0:00 httpd
8895 apache 15 0 6412 6412 6204 S 0.1 1.2 0:00 httpd
8917 apache 16 0 6332 6332 6120 S 0.1 1.2 0:00 httpd
8918 apache 15 0 6332 6332 6120 S 0.1 1.2 0:00 httpd
8935 apache 15 0 6352 6352 6140 S 0.1 1.2 0:00 httpd
8952 apache 16 0 0 0 0 Z 0.1 0.0 0:00 httpd <defunct>
8953 apache 16 0 0 0 0 Z 0.1 0.0 0:00 httpd <defunct>
8954 apache 16 0 0 0 0 Z 0.1 0.0 0:00 httpd <defunct>
8956 apache 16 0 6336 6336 6124 S 0.1 1.2 0:00 httpd
8959 apache 15 0 0 0 0 Z 0.1 0.0 0:00 httpd <defunct>
8960 apache 16 0 0 0 0 Z 0.1 0.0 0:00 httpd <defunct>
1 root 15 0 468 428 412 S 0.0 0.0 0:03 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 keventd
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 kapmd
4 root 34 19 0 0 0 SWN 0.0 0.0 0:02 ksoftirqd_CPU0
5 root 15 0 0 0 0 SW 0.0 0.0 0:15 kswapd
6 root 25 0 0 0 0 SW 0.0 0.0 0:00 bdflush
7 root 15 0 0 0 0 SW 0.0 0.0 0:00 kupdated
8 root 16 0 0 0 0 SW 0.0 0.0 0:00 mdrecoveryd
12 root 15 0 0 0 0 SW 0.0 0.0 0:00 kjournald
136 root 15 0 0 0 0 SW 0.0 0.0 0:00 kjournald
137 root 15 0 0 0 0 SW 0.0 0.0 0:00 kjournald
138 root 15 0 0 0 0 SW 0.0 0.0 0:16 kjournald
139 root 15 0 0 0 0 SW 0.0 0.0 0:00 kjournald
482 root 15 0 0 0 0 SW 0.0 0.0 0:00 eth0
574 root 15 0 588 536 500 S 0.0 0.1 0:00 syslogd
579 root 15 0 444 428 384 S 0.0 0.0 0:00 klogd
599 rpc 15 0 588 580 572 S 0.0 0.1 0:00 portmap
627 rpcuser 15 0 780 776 760 S 0.0 0.1 0:00 rpc.statd
739 root 15 0 464 412 412 S 0.0 0.0 0:00 apmd
793 root 15 0 856 744 704 S 0.0 0.1 0:00 sshd
826 root 15 0 880 760 688 S 0.0 0.1 0:00 xinetd



they appear to stay defunct. I've never seen this before untill today.

Try restarting apache or just reboot the server.

are ALL http processes defunct or just some of them?

You may want to check out your modules, ie php and such - did you change servers/binaries/update software lately?

I know that this topic is so damn old... but i got exactly these problems since today - after a kernel upgrade on RH 7.3 via up2date
Anyone else noticed this/has any problems with that?!
I mean the only thing which was updated was the kernel and the quota package. Not apache, php or anything else...
I also restarted apache & mysql to see if this brings any improvement, but no. Than i rebooted, but this didn't help either.

So any suggestions on this?
I just don't get it...
Maybe the standard RH kernel is faulty?

http://www.webhostingtalk.com/showthread.php?s=&threadid=79772

What kernel version and what version of apache are you using? Are you using the Red Hat rpm of apache, or did you build it?

BTW: You still could have started your own thread. :)

Thanks for the explanation, i do know what the "defunct" and zombie etc. stuff means, but I don't get why this happens.
There's enough free RAM, the system itself looks healthy etc. etc.
I do use Redhat's RPM apache Version. (1.3.27) and just updated to the newest kernel rpm via up2date (2.4.20-18.7)

I don't use the rpm for apache and have had no problems with the new kernel and apache on my test machine. Maybe someone else here using red hat's apache rpm will be able to duplicate your problem. You might try getting the apache source rpm and rebuilding it. Sorry I wasn't of any help, but I don't use their apache rpm.

the idea with the src rpm was a good one, unfornately it didn't resolve the problem! :/
damn. this sucks!

regards,

I upgraded the kernel and didn't notice any issues. Having some defunct processes are fine. If you are worried, just restart the server

nahh. but not that many and not if the server is really healthy.
I nailed the problem down to the apache file size limit...
[Sun Jun 8 14:17:04 2003] [notice] child pid 7566 exit signal File size limit exceeded (25)
[Sun Jun 8 14:17:04 2003] [notice] child pid 7565 exit signal File size limit exceeded (25)
[Sun Jun 8 14:17:04 2003] [notice] child pid 7564 exit signal File size limit exceeded (25)

how can i compile apache with the right settings?

Thanks again!

Thanks all for your help....
I finally nailed it down to the access_log which was 2GB big, which is the limit of the file system!

Guess i should implement some logrotation :D

Thanks everyone for their suggestions/help!

regards,

Originally posted by BiGWill
Thanks all for your help....
I finally nailed it down to the access_log which was 2GB big, which is the limit of the file system!

Guess i should implement some logrotation :D

Thanks everyone for their suggestions/help!

regards,

Ahh you know what, the same thing happened to me exactly a few months ago. Logs weren't being rotated and eventually filled up the hd. All httpd processes became defunct, though the serving of pages didn't seem much affected. Finally noticed the problem and got things fixed.

Originally posted by BiGWill
I finally nailed it down to the access_log which was 2GB big, which is the limit of the file system!

Doh! :homer:

Originally posted by GeorgeC
Ahh you know what, the same thing happened to me exactly a few months ago. Logs weren't being rotated and eventually filled up the hd. All httpd processes became defunct, though the serving of pages didn't seem much affected. Finally noticed the problem and got things fixed.

you sure?
it did affect the serving of pages heavily. that's why i wonder...

So on a cPanel server - I could just delete the domlog that's too big right?

Since it' can't be written to - then the information in it would have already been parsed into a stats prog? (awstats/webalizer)