I have a customer who has been getting ddos'ed all week, he has a dedicated server with me and the incoming crap has exceeded 36 mbits. Hardcore DDOS.

Blocking the ips at the server level is useless, when GNAX started blocking IPs upstream from us they ran out of ACL room (at least count over 2300 ips involved, yes 2300).

So we've null routed the ip and ... now what?

I mean wtf can you do when you have more than 2000 ips lining up to ddos you capable of producing this kind of throughput?

What kind of traffic is it?

Holy crap. One of my servers just had a 10mbit dos attack for a few hours, I can't even begin to imagine a 36mbit DDOS.

Sorry I don't have anything useful to say, but I know adultwebmasterinfo.com went through a DDOS attack more than double that. Might wanna ask Tim.

Originally posted by innova
What kind of traffic is it?

Straight to port 80, it creates a 408 in access_log for the server (not a vhost log) so that leads me to believe it's going straight to port 80 on the ip, here's the 408 extracts from access_log and the unique ips (warning the ddos.txt is over a meg):

www.dixiesys.com/ddos.txt
www.dixiesys.com/ddos_ips.txt

The ddos.txt isn't 100% complete since after the httpd processes are all tied up the ips just tie up the tcp stack and nothing gets served. So if an ip shows up 1 time it could have sent hundreds of requests but only 1 or 2 ended up in the log, ditto for the ip list, there were I'm sure other ips ddosing but NOT showing in access_log due to never actually being able to connect to httpd. And this still shows 2500 ips! I wouldn't be at all surprised if the real # of ips was quite a bit more.

I'm working on a script to parse these ips and send bitch letters to the abuse emails for the management of the individual class-A it is part of (sigh I could really use this time doing something else).

Go as upstream as you can, and ask the provider to block data mathching the type that is attacking you.
It could be blocking HTTP for a week, but it could save you hundreds, if not thousands of dollars in bandwidth charges.

*edit*
Also, to minimize the effect on you, you can also parse all those IP's trying to access the apparent bogus data into your firewall.
Dont send a reject, just straight out drop the packet at your firewall.
It defenitely wont stop the attack, but itll save some bandwidth in trying to send replies.

DDOS sucks totally i got hit with 230mbit before, the datacenter charged me 75 dollars for clean up.

From a customer point of view, how about balancing his website on 2 servers? There is a risk you'll be attacked on the two servers but his website can be up.

Best of luck!

I've actually figured out a way to use zoneedit and multple servers to put his web site on one server and hide it behind any number of "listeners" using the failover dns from zoneedit. The problem is will the customer pay for this sort of thing because each listener requires VPS (maybe haven't tried compiling the app on VPS yet) or a full server as it runs a daemon on port 80 and then plays middle man like a reverse proxy and the browser never actually "touches" the real server ip.

I actually have a test running:

http://209.51.150.79/

This is a copy of the proxy running on my backup system, there is no http on this server, no files for my forum, nothing. My forums are actually on another server. So far the tests are successful, luckily the site in question is a basic php driven site nothing real oddball so this basic proxy setup would work fine. (so far). Also forums is on another subnet alltogether, meaning you could run this app on a server in, say, EV1 and pull the data from the "real" server in GNAX. So this customer is likely to be on the prowl for some cheap dedicated servers soon, and I would think this daemon isn't gonna need major horsepower, a low-end P2 (heck even a cobalt raq4) server should handle a listener easy.

Oops haha I put the REAL ip of my forums, so fat lotta good that link is doing showing the proxy in action.

Actually I just got it working on a VPS from Adiungo so you can try it out instead:
http://swi.mysite.adiungo.com/

1) Install apache module called mod_dosevasive to prevent DOS attack.This will make sure that apache doesnt crash

2) Install apf (firewall ) which helps preventing DOS attack....

3 ) use this bash script ( execute it )
--------------------------
#!/bin/bash

/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe ip_conntrack_ftp

rm /root/.dyn*

echo "Setting kernel tcp parameters to reduct DoS effects"

#Reduce DoS'ing ability by reducing timeouts
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 1800 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
echo 0 > /proc/sys/net/ipv4/tcp_sack
echo 1280 > /proc/sys/net/ipv4/tcp_max_syn_backlog

#ANTISPOOFING
for a in /proc/sys/net/ipv4/conf/*/rp_filter;
do
echo 1 > $a
done

##
#NO SOURCE ROUTE
for z in /proc/sys/net/ipv4/conf/*/accept_source_route;
do
echo 0 > $z
done
#SYN COOKIES
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
#echo $ICMP_ECHOREPLY_RATE > /proc/sys/net/ipv4/icmp_echoreply_rate
echo "1" > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses
echo "1" > /proc/sys/net/ipv4/conf/all/accept_redirects
echo "1" > /proc/sys/net/ipv4/conf/all/log_martians

# NUMBER OF CONNECTIONS TO TRACK
echo "65535" > /proc/sys/net/ipv4/ip_conntrack_max

# Set default policies
/sbin/iptables -P INPUT ACCEPT
/sbin/iptables -P OUTPUT ACCEPT
/sbin/iptables -P FORWARD DROP

/sbin/iptables -F
/sbin/iptables -F INPUT
/sbin/iptables -F OUTPUT
/sbin/iptables -F FORWARD
/sbin/iptables -F -t mangle
/sbin/iptables -X

/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A INPUT -d 127.0.0.0/8 -j REJECT

/sbin/iptables -A INPUT -i eth0 -j ACCEPT

/sbin/iptables -A INPUT -m state --state INVALID -j DROP

### chains to DROP too many SYN-s ######
/sbin/iptables -N syn-flood
/sbin/iptables -A syn-flood -m limit --limit 100/second --limit-burst 150 -j RETURN
/sbin/iptables -A syn-flood -j LOG --log-prefix "SYN flood: "
/sbin/iptables -A syn-flood -j DROP
-------------------------------------------------------

Ooh thanks a few of these I haven't seen mentioned before. I will definitely add this to my pre-firewall rules for APF and to the rc.local for the proc/ stuff.

Originally posted by Dixiesys
I have a customer who has been getting ddos'ed all week, he has a dedicated server with me and the incoming crap has exceeded 36 mbits. Hardcore DDOS.

Blocking the ips at the server level is useless, when GNAX started blocking IPs upstream from us they ran out of ACL room (at least count over 2300 ips involved, yes 2300).

So we've null routed the ip and ... now what?

I mean wtf can you do when you have more than 2000 ips lining up to ddos you capable of producing this kind of throughput?

36Mbs DDOS? Be lucky it was only 36Mbs for a DDOS, hardly hardcore by DDOS standards.

I've never seen upstream providers ACL/null the source IP's in a DDOS, that is unusual, most of them will just null the IP under attack at the edge/ingress/peer points on their backbones.

We had a customer (frequents here daily) - get hit for a week on and off by a DDOS ranging from 350Mbs to 1.3GBs. Last track we had over 750K source IP's logged, and gave up and had to wait it out, move customers box around, etc - they then started hitting his dns zones instead of IP directly.

Oh, then they started attacking other ISP's and forging source IP's using this customers IP space. It was a fun week to explain to the masses of clueless companies (ISP, hosting, etc) that it was a forged IP and not US or the CUSTOMER attacking their servers.

Sigh, script kiddies, wish they would put their talents to better use, heck, I'd even hire some of the talented ones as coders for our company and put their skills to good use!

Tom

Originally posted by blessen
1) Install apache module called mod_dosevasive to prevent DOS attack.This will make sure that apache doesnt crash




please note that

mod_dosevasive

will break frontpage if any of the sites on the server are using it.