One problem after another, and my admin is offline. :angry:

I go to update my site, and FTP times out. I go to visit my site and the page loads, but very slowly.

I log into SSH (slow, again) and check top


10:28pm up 1 day, 3:03, 1 user, load average: 119.84, 106.30, 85.30
437 processes: 417 sleeping, 20 running, 0 zombie, 0 stopped
CPU states: 76.8% user, 23.1% system, 0.0% nice, 0.0% idle
Mem: 506236K av, 500804K used, 5432K free, 0K shrd, 2760K buff
Swap: 1048280K av, 596680K used, 451600K free 43440K cached


not good. apache is the only user and httpd the only command that I see.

netstat -ta

pretty well every connection is from a different source, and slightly more are TIME_WAIT than ESTABLISHED

My server is grinding to a halt, and I dont know what to do. Stuff like this always seems to happen during the rare instance that my admin is offline. It's either a conspiracy against me, or Murphy's law.

Anyways, any insight or help would be greatly appreciated :)

=THAnks

I would stop Apache, for about 10min then start it again. If someone is DoSing you, they may give up.

That's an idea, but one would think that after several hours of unsuccessful pounding, the culprits would give up (if it is infact a DOS)

I'd rather solve this problem with something other than psychology :)

Well... are they repeated attempts from the same IPs?

If not, it could be a site on your server just got popular.

You could always filter the IPs with IP Tables... that would be a pain though.

You could also kill some apache processes, it'll help for a little bit.

What's happened is Apache is sucking up all of your RAM, that doesn't help your load average. The lack of RAM hinders other processes from starting/forking, e.g. SSH or FTP.

No, they arent repeated attempts from the same IP.

It is possible that one of my sites has become popular, but my server should be robust enough to handle it. I was pushing 16 megabit with a similar server, and the load averages were nowhere near as high. Right now im not even doing 1/10 of that.

If you are running Portsentry (most cPanel/WHM systems do) stop it and then start it again.

Hmmm, odd... don't know what to tell you. :) Something is causing Apache to eat up RAM, aside from people accessing the server. Do any of the sites use PHP or mod_perl?

Sorry I can't be of much help, I just think a site got popular. Wait, what do your access logs show?

no port sentry. I'm running Plesk.

My site uses php, but not to the extent that it would cause problems.

Did you tried with
RLimitMEM
RLimitCPU ?

how about ps -aux ?

or even

ps aux | grep apache

or

ps aux | grep user

etc etc....

same here. Apache 1.3 got overload and crash with 60 TIME_WAIT simultany connections.
The thing is strange that I tell it in the configuration that after 40 connection it must refuse other one.
Perhaps Apache 2.0 will correct that ?!

Hy,

enable the server-status handler in your Apache config file, kill -HUP your apache and check the server status page with http://your.domain.com/server-status.

There you can see which apache process with which request eats up the memory and system power.

After checking dont forget to disable the server status handler and to restart the apache.

Greetings
Oliver

excellent. I'll keep all this in mind if it happens in the future. I ended up just stopping apache ofr a minute or two, and restarting it. everything was fine after that.

thanks for all the help, I really appreciate it.

Hrm,

A little while back one of the servers I admined got DoSed on port 80 (Apache). Ended up setting up a script that did a netstat -an every minute, found an IP with more than 5 active connections and dynamically added the iptables rule.

I would post the script but I seem to have since lost it. :( In either case it worked relatively well and while some clients were blocked accidently we (the Web Host and I) certainly thought it was the smartest move rather than have all 300 domains on the server down.

Cheers,

Stuart

You can try this patch ReadRequestTimeout-directive (http://www.apache.org/dist/httpd/contrib/patches/1.3/apache_1.3.22-ReadRequestTimeout-directive.patch)


Allows specification of a ReadRequestTimeout, so servers with a large
Timeout setting can still get rid of clients not sending requests.


Jimmy

I suggest you check out user cgi files to make sure they are OK (it's bad that you didn't run ps -aux, but anyway, if it was a php it would be hard to trace)... maybe it was a buggy cgi program/script (infinitive loop etc)... sometimes it happens :(

Originally posted by Perlboy
A little while back one of the servers I admined got DoSed on port 80 (Apache). Ended up setting up a script that did a netstat -an every minute, found an IP with more than 5 active connections and dynamically added the iptables rule...

We use a similar script, only it notifies us instead of blocking the IP address. Typically a decent (if there is such a thing) DOS attack will trigger a notification when the server load goes too high, as well.

Most DOS attacks are going to be SYN floods. Our script doesn't check for active connections, but only requests for connections which would better indicate a SYN flood. There is no point in creating active firewall rules for a SYN flood, as the IP addresses will probably be spoofed.

If you are going to create active firewall rules, you would need to check for "active connections" like you said, as they would then be using a valid IP address that you could block. However, a lot of browsers will open more than one connection, so I would recommend a much higher number than five. Still, this isn't going to be the majority of your DOS attacks, but they are the easiest to deal with.

However, woodsheds problems didn't indicate a DOS attack, but only an available/overloaded web server that was receiving requests.

Originally posted by bitserve
However, woodsheds problems didn't indicate a DOS attack, but only an available/overloaded web server that was receiving requests.

That's a possibly wrong assumption.


netstat -ta

pretty well every connection is from a different source, and slightly more are TIME_WAIT than ESTABLISHED


This indicates it could possibly be a DDoS attack on the Apache port 80 from spoofed IP addresses or even slaves. The fact the server loads are massive indicates huge spawning of Apache children.

While it IS possible he is receiving mass traffic, all things considered, the possibility of this causing huge server loads compared relatively to a DDoS is low. How many of us have extremely busy websites (ie. 100's or 1000's of hits a second). I have one website which cops 110,000 hits a day (all of which run through a Perl script) and never see loads above around 0.5 - 1.

Cheers,

Stuart

Originally posted by Perlboy
That's a possibly wrong assumption.

Possibly wrong, yes. But most likely it was a correct assumption. The indicators were not consistent with a DOS attack.

This indicates it could possibly be a DDoS attack on the Apache port 80 from spoofed IP addresses or even slaves.

It does not indicate any such thing. In general, you can not spoof a source IP address and have the packet make it back to you in order to complete a TCP connection and have a TIMED_WAIT. A man in the middle attack would allow this, but is not conceivable for multiple source addresses from multiple networks (DDOS).

The fact the server loads are massive indicates huge spawning of Apache children.

It only indicates the inability for the server to handle the current requests, resulting in apache queueing the requests, which results in a high server load. 417 out of 437 processes sleeping more sounds like the inability to fork any additional apache processes. He probably had a run away apache process.

While it IS possible he is receiving mass traffic, all things considered, the possibility of this causing huge server loads compared relatively to a DDoS is low.

Agreed that it wasn't a problem with massive traffic, but it wasn't a DOS either. It was more than likely a malfunction.

Next time (hopefully there won't be a next time) use CJCS's advice and look to see if its a certain site thats getting all the hits and look for any sort of pattern in the requesting IPs. Sometimes this can just be that someone had an endless loop or included a file that included itself etc into a loop, I've seen it before.