Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Apache hangs when having lots of ..reading.. requests

[ovisopa]

Hello,

A weird thing happents once on few days on my server, and it's not a regular thing or on exact time.

When I have this problem no page can be loaded in the browser, but WHM is working, i think because it's accessed by IP .

In apache status page I can see lots of ..reading.. requests which are there even for 10 - 20 seconds sometimes.

Usualy the server has 10 - 20 requests/s :

CPU Usage: u35146 s2297.05 cu2.74 cs4.5 - 6.97% CPU load
11.2 requests/sec - 83.0 kB/second - 7.4 kB/request
10 requests currently being processed, 8 idle servers

But when the ..reading.. requests appear it goes much higher like 100 to 200

11.5 requests/sec - 85.3 kB/second - 7.5 kB/request
200 requests currently being processed, 0 idle servers

after 3 minutes :

160 requests currently being processed, 30 idle servers

When I logged on to SSH I saw that there are ~150 conection from a single IP .

Can somebody tell me what should I do about those ..reading.. requests which take alot of time and sometimes causes apache to not deliver other requested pages

Thank you .

[cybercast]

Hello,

I have been there. This is a denial of service. It is probably someone attacking one your customers website. My first suggestion would be blocking the IP at network level. Detect who is attacking. It could be a single IP or it could be several IPs. This could also be called a distributed denial of service. Try to find if there is a vulnerability at your apache and patch it or upgrade it. A good IPS should also block the attack so ask your service provider. Good luck.

[ovisopa]

10x for the answer.

I have a dedicated server at DedicatedNow.
When I saw this thing first time my admin blocked one IP from APF

/etc/apf/apf -d xxx.xxx.xxx.xxx

but I see that everytime there are different IP's

I will ask my admin about the apache patching because I don;t have a clue on how I can find if it's vulnerable. Is there an online tool to test ?
I have Apache/1.3.37 , PHP Version 5.2.1 and MySQL Versiune server: 5.0.45-community-log

Thank you.

About the maximum number of conections to the server I told my admin to increase it to 200 because before was 100, is it ok to increase it to 300 ? (My server specs are Dual Intel(R) Xeon(TM) CPU 3.20GHz ; 2GB ; 2x WDC WD2000JD)

[cybercast]

Then you have a DDoS (Distributed Denial of service). Maybe it is not a big one but I can tell you it could get worst. I suggest you to ask your datacenter if they have an IPS (intrussion prevention system) as that will help. Explain them the situation.

When it comes to ddos it is hard to block the attack. First thing would be patching the server with latest releases. Block the attack at IPS. If you have IPS and the attack persist then block all access to the IP attacked. Find who is atracting the attack and get rid of him.

[ovisopa]

In the mean time I though the problem was solved by activating APF DoS protection and installing mod_evasive (mod_security is already instaled) but now in the last 2 days it started agian

I asked my datacenter and they seems not to have any protection against DoS ( Cisco Guard DDOS Protection, Tipping Point IPS/IDS Protection )

We are currently looking at several possible anti DDoS solutions. We do not yet have a date when these will be active in the network.

What do you suggest to do next? .. I just cant move again to a new datacenter, in the last few years, almoust yearly I moved my websites, from shared to a better shared till I had to many clients and to big websites to stay on that shared .. than moved on DedicatedNow , dedicated server Dual Xeon CPU 3.20GHz, 1 GBDDR .. now few months upgraded to 2GB and for 3 months everything was very good 100% uptime everymonth.

Now I have this issue with DoS.

In SSH is there any way to "see" the attacker ?

because I'm not usre this is showing me the attacker too :

netstat -plan|grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

I can see lots of conections .. ~ 150 from a single IP sometimes, but this happents almoust all the day and the server is running nice, the problem appears when I can see in apache status many ..reading.. conections which I don't know how to identify and eventualy block

Any further help will be appreciated.

Thank you.

[cybercast]

My suggestion is to find and suspend the client that is the target of the attack. Then block the IP at router level and if possible at carrier level. That is the best you can do at this point. Isolate the problem.

[ovisopa]

yes but this is kinda hard for me because I'm not to good at SSH.

Is there any SSH command to show me all the conections (IP of the source) and target account on my server, because I have almoust all accounts on the same IP. I have just 2 website which have unique IP, and the others are all on the same shared IP.

In the last period i'm not facing this problem, and I don;t know if it's because of mod_evasive or because they stopped attacking me. About mod_evasive I'm sure it's working because it blocked a bage in my admin panel which used like 30 istances of FCKEditor in the same page. In that page I was see ing only the first 2 iframes of FCKEditor and all others ware just blank.
After adding my IP to mod_evasive allow list it worked again

I'm not sure yet why it doesn't sent emails or create any log when it blocks something, because it was setup to .. but didnt get any till now.

[cybercast]

in order to verify who is the target you may activate "server-status" at your apache configruation file. make sure to activate the extended server status. Then you will be able to verify the connection and find the target.

Also watchout the logs of mod_evasive as your disk could get full by them.

[Tim Greer]

It doesn't sound like a large attack anyway, so using tools like netstat with various commands, such as you've pasted above, are what will usually tell you the most. The server status is helpful, and scoreboard. You could have a cron or script run to check and detect and block attacking IPs, and you can rate limit (iptables software firewall) how many times an IP can hit the site in what amount of seconds, so it could start blocking, dropping so your request limit isn't hit and blocking legitimate traffic/requests.

Upping your max connections to 300 is fine, your server can handle it, but it also depends on the wait/keep alive times, timeout settings, etc., the script (if any), any persistent connections or resulting database queries if it's a hit on a script that launches them. mod_evasive/mod_security aren't going to do much, especially mod_security. A lot of times a DDoS is a lot of sources making legitimate requests. You can sometimes catch the attack signature in the logs or break down the packets to get it, but once the hit is made, it's made and the same IP probably won't hit it again for a long time, even if it hit it 200 times in a single minute. Sometimes it is small enough where you can block the hardest hitting IPs and you'll be fine, until they find another compromised system to launch attacks from to your server.

Ideally, you will want to have a site per IP or a small number of sites per IP, and none on the main system IP, then if it becomes too much, you can find the target site (that and looking at the logs and scoreboard, etc.) and rate limit incoming connections to that site's IP, or even just drop (or reject, depending on the type of attack) any requests to that IP and wait it out and try again. This way it won't affect the other sites hosted on the other (virtual) IPs.

Just remember that the more IPs added to iptables, as the chains and rules grow, it can seriously consume a lot of memory and backfire, so if it's a massive attack with hundreds of thousands of IPs, you might have to get it null routed (or just request incoming requests if your DC doesn't have that ability for you to request for some weird reason). Again, being legitimate requests a lot of the time, so often (again, depending), the anti-DoS/DDoS services might not help anyway (and often some use a proxy type of filter that just makes it impossible to control and track on the server end if it continues). There are too many variables to just say what exactly to do, but it doesn't seem too bad of an attack and it might a busy site with a lot of loading, too.

[ovisopa]

Today happent again for a long period (10 minutes), until now, after my admin installed mod_evasive it happent again but after ~1 minute the number of request was droping and I thought that mod_evasive blocked it
(i have a "problem" about mod evasive, I cant see logs and I don;t get emails about its actions, my admin configured the email address and the log file but it's not working)

Today when happent, i blocked 3 IP's but I'm not sure this was the cause. Anyway after I bloked them it stoped. After one hour I unblocked 2 of them because I saw it ware visitors from my city.

For exemple, now as I write this post I logedon to SSH and used this command :

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

i got a list and the bottom of it it was this :

53 85.204.x.XX
65 82.144.2x.XX
79 89.35.1x.XX
81 85.186.20x.XX
83 86.122.4x.XX
85 194.102.10x.XX
99 82.127.2x.XXX
118 86.122.4X.XX
847 91.195.x.XX

the number in front is the number of conections ? this is what I knew.(Please corect me if I'm wrong) anyway now I can see a huge number there, but the server is not stressed at all (CPU usage is 0.57) and in WHM apache status page i have this :

Current Time: Wednesday, 16-Apr-2008 05:47:41 EDT
Restart Time: Friday, 04-Apr-2008 06:57:15 EDT
Parent Server Generation: 6
Server uptime: 11 days 22 hours 50 minutes 26 seconds
Total accesses: 10980173 - Total Traffic: 71.7 GB
CPU Usage: u4028.26 s318.66 cu.17 cs.31 - .421% CPU load
10.6 requests/sec - 72.8 kB/second - 6.8 kB/request
17 requests currently being processed, 10 idle servers
GWWRWRWG___G_W_W_RW_G_R__.G..R..................................
................................................................
................................................................
................................................................

So no problem. even if the number of conections reported by netstat is huge.

I'm not sure what to think, and if it is an DoS, and what type of DoS because i'm sure there are different ways. I used a program for testing purpose, the name was something with DoS atack .. or something like that but in the time of attack I was able to see the source Ip and target website

When the thing with RRRRRR happents I can't see anything, nor the IP or target website or IP.

On my server the most website are on one IP (100 websites), the bigger websites I have, are using a single IP/website but I only have 2 websites which have different IP's (I'm using 3 of 5 available IP's on my server)

[cybercast]

get ddos mitigation service for that domain. isolate the problem.

[ovisopa]

Today I had again that problem (

I attached another printscreen, about the huge amount of dots from WHM Apache Status page ( "." Open slot with no current process ) is it normal to have so many there? Can that be reduced, it will save some resources .. it will help ?

About the "top" .. can you suggest anything that may help ?

Is there a company which can optimize my server to be more safer just by intalling / tunning the software on my server ? whitout any hardware change ?

Thank you.

[andren]

You are *not* target of a dos attack. This pattern looks like a normal load issue, load gets higher and apache takes longer to process the requests.

Your screenshots don't tell much, there is a 4 hour time difference between the two. The dots are slots for connections to apache (that number is too high btw).
Mod_evasive with good rules might help. The netstat numbers are high, though it doesn't tell us the connection status, which might be important.

Also get a competent sysadmin.

[cybercast]

get us the extended version of your server-status report. So we can get more information.

[ovisopa]

By extended status you mean the file in attachement ?

If not please tell me what exactly should I show you.