KNL-BSW
08-09-2004, 09:35 PM
We have a customer who is getting hit with valid GET requests from spoofed IP's.
We are talking about over 200 requests even when the site is down to the IP address.
When the site is up it get hits so hard that it brings apache to its knees and stops serving pages.
We have installed and configured mod_dosevasive, didn't help as it is using multiple random spoofed IP's and using valid GET requests.
Raised ServerLimit to 1024 and raised MaxClients to 768 in hopes of at least being able to allow the site to stay up, didn't help.
Spoke with GNAX about it and with a full netstat/grep they could not locate enough significant of any one single IP to prevent it at the router.
Does anyone have any ideas on how to protect against these?
Thanks,
Crucial
08-09-2004, 10:01 PM
Have you looked at or tried snort? Also a good idea when blocking spoofed ips is not the ips themselfs is the ports there coming from and the ip or port there attacking.
if you had 1000's of spoofed ips on 129.1.1.1 source ports ranging from 1000-4000. I would filter......
any 1000-4000 to HOSTING.IP HOSTING.PORT
If the hosting port is a shared ip then it might filter out good traffic but this is one way I do this at router level and even server level "ipfw via freebsd"
KNL-BSW
08-09-2004, 10:08 PM
That is sort of what I was thinking. I shut the site down until we can resolve this. I didn't have a choice. The customer is on a dedicated IP and they are hitting port 80. They are hitting valid "GET" requests. This started after we started using mod_dosevasive. Prior to this it was just the main index.php that was pulled. The minute we started using mod_dosevasive they started pulling a page with a get command. Renamed the page and they picked another one.
I'm baffeled as how to stop it. :(
datums
08-09-2004, 11:23 PM
Hello Larry
How many request per minute would you say ?
So would you say the attack is ip based ?
connect to 1.2.3.4 get /index.php
KNL-BSW
08-09-2004, 11:30 PM
Oh, over 200+ when the site is up. And it's not at the IP. They are using DNS to resolve and attacking domain.com/ It was first just domain.com/ which returned index.php. mod_dosevasive stopped that so they resorted to domain.com/filename.
Problem is they using spoofed IP's. It's pretty obvious too when you have 600 requests over a 2 minute period in the access log for the site all for the same file and all from the same OS/Browser. :( And this within 15 minutes after turning the site back on after having it down more than 48 hours.
datums
08-09-2004, 11:42 PM
Instead of bringing the site down, change the A record temp to 127.0.0.1 the zombie machines will hose themselves.
I know this is not a solution, but as you continue to brainstorm.
How big is the site being attacked? does it have db ?
If it's only webpages, you might want to add the domain to another server, distributing the attack across servers.
KNL-BSW
08-09-2004, 11:50 PM
It's a fairly large database driven site. Goes through about 100GB of bandwidth monthly without this problem. I've already been thinking of dropping them on a seperate server with httpd restart scripts until we find a viable resolution.
Tried the DNS thing. :( Didn't work.
Crucial
08-09-2004, 11:54 PM
Whats the OS on the server and do you have router access for ACL's? The solution I posted will work 100% you will need to tcpdump -n on the machine and find out what source ports and destination ports there using first. Usally they will be around the same if there using the same program to generate the attack.
datums method will not work if the attacker is attacking ip address vs. domain name isnt that correct?
datums
08-10-2004, 01:17 PM
cord that is correct If it is ip based then, changing the domains ip would help
If it is domain base changing the ip to 127.0.0.1 on temp basis will hose the zombies that are sending the request.