Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Need help, DDoS attack on my VPS.. down for 2 days now

[Phatmat]

Hey guys, I'm in need of some help. For the past two days there has been a DDoS attack on my VPS.

My VPS specs are fairly small, so it is easy to take down, with only 512mb RAM, and a 666MHz CPU. I'm running the latest CentOS.

I've used the command:
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | sed -e s/'::ffff:'/''/g|cut -d: -f1 | sort | uniq -c | sort -n

To check the IP's that are hitting me, and there seems to be hundreds of different ones from all different countries.

" 26 99.147.202.54
28 142.59.192.124
29 119.160.178.45
29 67.206.209.63
36 202.27.218.72
47 66.75.49.158
67 87.3.160.142
130 80.117.212.205"

" 30 70.153.64.140
30 85.94.123.78
38 82.249.18.116
39 66.75.49.158
51 190.213.16.4
51 80.54.48.5
79 87.11.54.124
116 87.3.160.142"

" 36 121.96.116.63
38 85.94.123.78
46 70.153.64.140
59 80.54.48.5
69 190.213.16.4
139 87.11.54.124"

" 44 81.164.95.51
52 190.213.16.4
57 76.3.94.140
109 219.93.18.98
186 82.117.202.46
187 189.127.141.70
208 212.156.145.206"

And the list never ends.. Every few minutes all the IP's change.


As you can see the IP's that are using up all my resources are random, and change about once every minute. I've tried adding the most resource consuming IP's to my iptables, with no luck - as more and more IP's will pop up with 200+ processes in use.

I've got (D)Dos deflate installed, and from what I can see it doesn't seem to be working..


I'm stuck here, what could I possibly do to get my site back online, with limited money resources? My host recommend that I try:

"Try with nginx as a reverse proxy and let us know how it works."

What is this, and how would I use this?

Any help at all would be highly appreciated,
Matt.

[khunj]

Can you try those commands, at least it will give an idea of what kind of flood it is :

# netstat -nt | grep ':80 ' | awk '{print $6}' | sort | uniq -c
# netstat -s
# dmesg | tail -n 20

And what kind of VPS : Xen, OpenVZ... ?

[Phatmat]

Here are the results:

[root@server ~]# netstat -nt | grep ':80 ' | awk '{print $6}' | sort | uniq -c
     20 CLOSE_WAIT
    195 ESTABLISHED
     33 FIN_WAIT1
     73 FIN_WAIT2
      1 LAST_ACK
    745 SYN_RECV
     13 TIME_WAIT

[root@server ~]# netstat -s
Ip:
    9256571 total packets received
    0 forwarded
    0 incoming packets discarded
    7732206 incoming packets delivered
    2343368 requests sent out
    881 dropped because of missing route
Icmp:
    1255 ICMP messages received
    704 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 1250
        timeout in transit: 5
    8 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 8
IcmpMsg:
        InType3: 1250
        InType11: 5
        OutType3: 8
Tcp:
    196 active connections openings
    38248 passive connection openings
    8081 failed connection attempts
    17107 connection resets received
    194 connections established
    7730325 segments received
    2313529 segments send out
    30366 segments retransmited
    360 bad segments received.
    148401 resets sent
Udp:
    2051 packets received
    8 packets to unknown port received.
    0 packet receive errors
    2051 packets sent
TcpExt:
    8081 resets received for embryonic SYN_RECV sockets
    153089 packets pruned from receive queue because of socket buffer overrun
    25193 packets dropped from out-of-order queue because of socket buffer overrun
    4 packets rejects in established connections because of timestamp
    63886 delayed acks sent
    15 delayed acks further delayed because of locked socket
    Quick ack mode was activated 9627 times
    503145 times the listen queue of a socket overflowed
    503146 SYNs to LISTEN sockets ignored
    65 packets directly queued to recvmsg prequeue.
    1448 packets directly received from backlog
    14583 packets directly received from prequeue
    330668 packets header predicted
    14 packets header predicted and directly queued to user
    581661 acknowledgments not containing data received
    166671 predicted acknowledgments
    14 times recovered from packet loss due to fast retransmit
    3323 times recovered from packet loss due to SACK data
    Detected reordering 35 times using FACK
    Detected reordering 32 times using SACK
    Detected reordering 17 times using time stamp
    41 congestion windows fully recovered
    53 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 189
    204 congestion windows recovered after partial ack
    1365 TCP data loss events
    TCPLostRetransmit: 2
    892 timeouts after SACK recovery
    268 timeouts in loss state
    4842 fast retransmits
    362 forward retransmits
    2648 retransmits in slow start
    6850 other TCP timeouts
    434 sack retransmits failed
    256042 packets collapsed in receive queue due to low socket buffer
    21335 DSACKs sent for old packets
    576 DSACKs sent for out of order packets
    2876 DSACKs received
    1 DSACKs for out of order packets received
    351 connections reset due to unexpected data
    12704 connections reset due to early user close
    483 connections aborted due to memory pressure
    1122 connections aborted due to timeout
    18 times unabled to send RST due to no memory
IpExt:

[root@server ~]# dmesg | tail -n 20
ip_conntrack: CT 1521102616: table full, dropping packet.

It is OpenVZ.

[yajur]

nginx is webserver it is best to install for this environment,but better to know to stop this issue

[Moonster]

Apparently your ip_conntrack table is full, you can review your table
with:

# cat /proc/net/ip_conntrack

The max number of connections is set in

# cat /proc/sys/net/ipv4/ip_conntrack_max

You can increase it with:

# echo "some number" > /proc/sys/net/ipv4/ip_conntrack_max

Which hopefully will help. looks SYN related.

[Phatmat]

Originally Posted by Moonster

Apparently your ip_conntrack table is full, you can review your table
with:

# cat /proc/net/ip_conntrack

The max number of connections is set in

# cat /proc/sys/net/ipv4/ip_conntrack_max

You can increase it with:

# echo "some number" > /proc/sys/net/ipv4/ip_conntrack_max

Which hopefully will help. looks SYN related.

What will this do? Allow more connections to the VPS?

I'm already out of RAM, so wouldn't this cause the DDoS attack to create more process, this make things worse?

I'm wondering, is there some way to prevent an IP having more than one process running? Some of the IP's (listed in my first post) have 200+ processes running, could I put a limit of say 10 here?

Let me know if my terminology is wrong, as from my current knowledge I think that the number next to the IP (in my first post) is the number of processes that that IP is using on my VPS.

- Matt.

[khunj]

A small SYN flood.
It's a bit weird your VPS cannot stand 700 half-opened connection.
Unfortunately, there's nothing you can do at the server level because you are using OpenVZ. It uses a single kernel, every users share it so you cannot tweak it (/proc/net, /proc/sys/net etc).
In the future, if you have to face SYN floods again, go for Xen VPS for instance, at least you could fight back

My host recommend that I try:
"Try with nginx as a reverse proxy and let us know how it works."

Unless there were some major changes to the TCP protocole last night and no one informed me, before a packet can reach the HTTP server backlog there must be a 3-way handshake sequence. During a SYN flood you are stuck (and will remain stuck) in the middle of it. It is a kernel problem, not an Apache/Nginx problem. The only help you could get is from your hosting company but... it looks like it's going to be tough

I think that the number next to the IP (in my first post) is the number of processes that that IP is using on my VPS.

If they are in the SYN_RECV state, they are just half-opened connections, no processes. They are just filling up your kernel backlog and when it's full, your server drops any further packet.

What about the 195 ESTABLISHED connections ? Is that the average/normal traffic on your server ?