Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Apache optimization, reach MaxClients every weeks

[osaris]

Hello,

I'm running a 3 Ghz Xeon with 2 GB RAM Dell server (PE 2850) which host 160 sites with Plesk. It make 120 Gb traffic per month.

I'm running FC3 with Apache 2.0.53.

Since 3 months, Apache crash apprimatively one time per week near 3 a.m. with a MaxClients reached error in error_log and the server load at 300 ! I'm unable to restart apache and I need to reboot the server with APC every time because shutdown command didn't reply !

Except when Apache crash, my server load is between 0 and 1 all the time.

Here is my apache config :

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 120

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2

##
## Server-Pool Size Regulation (MPM specific)
##
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 160
MaxRequestsPerChild 2000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I've already check if the problem was coming from a cron job but it didn't seems to.

Maybe upgrading to Apache 2.0.59 would resolve this strange problem ?

Regards,

Osaris

[Slidey]

what do the main apache logs show? can you graph number requests per minute/10 minute/hour period?

add a ps auxww every 5 minutes for an hour between 2 + 3 (maybe lengthen it by 30 minutes each side) and output to a file, check whats running

[osaris]

Thanks for your (very) quick reply.

The main apache logs didn't show special usage, still few call to the webmail but not lots of requests.

I've already captured the apache-status every minute before the crash and we see that in the scoreboard that all the slots are used by a process. I've try to add an apache restart at 3:08 a.m. but it can't kill all the processes ! It sends a SIGTERM then a SIGKILL on the 256 processes which are running.

I've also captured the ps auxww every minute between 2:45 and 3:15 but it only show that there's many apache processes, nothing more...

Thanks for your help

[Slidey]

im on call this weekend, so virtually have to sit in front of the pc anyway...

it strikes me as strange that it happens at exactly the same time every night.. even if your site got busy, it would/should have some variation.. is it a certain cgi, or a busy db, or even a db backup (we get all 3 sometimes, but more cgi issues than db as we run on seperate machines)

so theres no defunct processes, just a shedload of active apache procs? can try to upgrade to latest - i dont regularly read changelogs so dont know whats changed and how they'd affect you..

[osaris]

The problem appears approximatively one time per week :

[root@srv13 httpd]# cat error_log* | grep MaxClients
[Sat Oct 07 03:07:58 2006] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Sep 25 03:24:39 2006] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sat Sep 30 03:11:46 2006] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Thu Sep 14 03:18:25 2006] [error] server reached MaxClients setting, consider raising the MaxClients setting

Apache-status let me think it's not a site busy issue because many site are listed. I've already checked that it's not a backup or CGI issue.

In fact it's like if httpd wasn't killing processes so the slots are filled with many many processes and apache die...

Did you know how I can easily upgrade to httpd 2.0.59 on FC3 ? (rpm)

Regards,

Osaris

[cywkevin]

Try lowering the value for timeout from 120 to 60 Of course if all else fails you may have to accept that you need a better server and upgrade.

[osaris]

Originally Posted by pixelized

Try lowering the value for timeout from 120 to 60 Of course if all else fails you may have to accept that you need a better server and upgrade.

I've lowered my timeout but I really didn't think that the solution is to upgrade to a better server because i would have more problem between 5 p.m and 8 p.m. each day !