Results 1 to 7 of 7
  1. #1
    Join Date
    Aug 2008
    Posts
    37

    Need help, DDoS attack on my VPS.. down for 2 days now

    Hey guys, I'm in need of some help. For the past two days there has been a DDoS attack on my VPS.

    My VPS specs are fairly small, so it is easy to take down, with only 512mb RAM, and a 666MHz CPU. I'm running the latest CentOS.

    I've used the command:
    netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | sed -e s/'::ffff:'/''/g|cut -d: -f1 | sort | uniq -c | sort -n

    To check the IP's that are hitting me, and there seems to be hundreds of different ones from all different countries.

    " 26 99.147.202.54
    28 142.59.192.124
    29 119.160.178.45
    29 67.206.209.63
    36 202.27.218.72
    47 66.75.49.158
    67 87.3.160.142
    130 80.117.212.205"

    " 30 70.153.64.140
    30 85.94.123.78
    38 82.249.18.116
    39 66.75.49.158
    51 190.213.16.4
    51 80.54.48.5
    79 87.11.54.124
    116 87.3.160.142"

    " 36 121.96.116.63
    38 85.94.123.78
    46 70.153.64.140
    59 80.54.48.5
    69 190.213.16.4
    139 87.11.54.124"

    " 44 81.164.95.51
    52 190.213.16.4
    57 76.3.94.140
    109 219.93.18.98
    186 82.117.202.46
    187 189.127.141.70
    208 212.156.145.206"

    And the list never ends.. Every few minutes all the IP's change.


    As you can see the IP's that are using up all my resources are random, and change about once every minute. I've tried adding the most resource consuming IP's to my iptables, with no luck - as more and more IP's will pop up with 200+ processes in use.

    I've got (D)Dos deflate installed, and from what I can see it doesn't seem to be working..


    I'm stuck here, what could I possibly do to get my site back online, with limited money resources? My host recommend that I try:

    "Try with nginx as a reverse proxy and let us know how it works."

    What is this, and how would I use this?

    Any help at all would be highly appreciated,
    Matt.

  2. #2
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    433
    Can you try those commands, at least it will give an idea of what kind of flood it is :

    Code:
    # netstat -nt | grep ':80 ' | awk '{print $6}' | sort | uniq -c
    # netstat -s
    # dmesg | tail -n 20
    And what kind of VPS : Xen, OpenVZ... ?
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  3. #3
    Join Date
    Aug 2008
    Posts
    37
    Here are the results:

    Code:
    [root@server ~]# netstat -nt | grep ':80 ' | awk '{print $6}' | sort | uniq -c
         20 CLOSE_WAIT
        195 ESTABLISHED
         33 FIN_WAIT1
         73 FIN_WAIT2
          1 LAST_ACK
        745 SYN_RECV
         13 TIME_WAIT
    Code:
    [root@server ~]# netstat -s
    Ip:
        9256571 total packets received
        0 forwarded
        0 incoming packets discarded
        7732206 incoming packets delivered
        2343368 requests sent out
        881 dropped because of missing route
    Icmp:
        1255 ICMP messages received
        704 input ICMP message failed.
        ICMP input histogram:
            destination unreachable: 1250
            timeout in transit: 5
        8 ICMP messages sent
        0 ICMP messages failed
        ICMP output histogram:
            destination unreachable: 8
    IcmpMsg:
            InType3: 1250
            InType11: 5
            OutType3: 8
    Tcp:
        196 active connections openings
        38248 passive connection openings
        8081 failed connection attempts
        17107 connection resets received
        194 connections established
        7730325 segments received
        2313529 segments send out
        30366 segments retransmited
        360 bad segments received.
        148401 resets sent
    Udp:
        2051 packets received
        8 packets to unknown port received.
        0 packet receive errors
        2051 packets sent
    TcpExt:
        8081 resets received for embryonic SYN_RECV sockets
        153089 packets pruned from receive queue because of socket buffer overrun
        25193 packets dropped from out-of-order queue because of socket buffer overrun
        4 packets rejects in established connections because of timestamp
        63886 delayed acks sent
        15 delayed acks further delayed because of locked socket
        Quick ack mode was activated 9627 times
        503145 times the listen queue of a socket overflowed
        503146 SYNs to LISTEN sockets ignored
        65 packets directly queued to recvmsg prequeue.
        1448 packets directly received from backlog
        14583 packets directly received from prequeue
        330668 packets header predicted
        14 packets header predicted and directly queued to user
        581661 acknowledgments not containing data received
        166671 predicted acknowledgments
        14 times recovered from packet loss due to fast retransmit
        3323 times recovered from packet loss due to SACK data
        Detected reordering 35 times using FACK
        Detected reordering 32 times using SACK
        Detected reordering 17 times using time stamp
        41 congestion windows fully recovered
        53 congestion windows partially recovered using Hoe heuristic
        TCPDSACKUndo: 189
        204 congestion windows recovered after partial ack
        1365 TCP data loss events
        TCPLostRetransmit: 2
        892 timeouts after SACK recovery
        268 timeouts in loss state
        4842 fast retransmits
        362 forward retransmits
        2648 retransmits in slow start
        6850 other TCP timeouts
        434 sack retransmits failed
        256042 packets collapsed in receive queue due to low socket buffer
        21335 DSACKs sent for old packets
        576 DSACKs sent for out of order packets
        2876 DSACKs received
        1 DSACKs for out of order packets received
        351 connections reset due to unexpected data
        12704 connections reset due to early user close
        483 connections aborted due to memory pressure
        1122 connections aborted due to timeout
        18 times unabled to send RST due to no memory
    IpExt:
    Code:
    [root@server ~]# dmesg | tail -n 20
    ip_conntrack: CT 1521102616: table full, dropping packet.

    It is OpenVZ.
    Last edited by Phatmat; 10-04-2009 at 10:29 PM.

  4. #4
    Join Date
    Nov 2007
    Location
    India
    Posts
    843
    nginx is webserver it is best to install for this environment,but better to know to stop this issue
    HostNotch Hosting Services 99.9% uptime Shared Hosting, Reseller Hosting
    yajur | Sales Team
    CPanel Hosting • R1 Soft • Offsite-Backup • Great Uptime
    http://hostnotch.com sales @ hostnotch.com

  5. #5
    Join Date
    Oct 2009
    Posts
    58
    Apparently your ip_conntrack table is full, you can review your table
    with:

    # cat /proc/net/ip_conntrack

    The max number of connections is set in

    # cat /proc/sys/net/ipv4/ip_conntrack_max

    You can increase it with:

    # echo "some number" > /proc/sys/net/ipv4/ip_conntrack_max

    Which hopefully will help. looks SYN related.

  6. #6
    Join Date
    Aug 2008
    Posts
    37
    Quote Originally Posted by Moonster View Post
    Apparently your ip_conntrack table is full, you can review your table
    with:

    # cat /proc/net/ip_conntrack

    The max number of connections is set in

    # cat /proc/sys/net/ipv4/ip_conntrack_max

    You can increase it with:

    # echo "some number" > /proc/sys/net/ipv4/ip_conntrack_max

    Which hopefully will help. looks SYN related.
    What will this do? Allow more connections to the VPS?

    I'm already out of RAM, so wouldn't this cause the DDoS attack to create more process, this make things worse?

    I'm wondering, is there some way to prevent an IP having more than one process running? Some of the IP's (listed in my first post) have 200+ processes running, could I put a limit of say 10 here?

    Let me know if my terminology is wrong, as from my current knowledge I think that the number next to the IP (in my first post) is the number of processes that that IP is using on my VPS.

    - Matt.

  7. #7
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    433
    A small SYN flood.
    It's a bit weird your VPS cannot stand 700 half-opened connection.
    Unfortunately, there's nothing you can do at the server level because you are using OpenVZ. It uses a single kernel, every users share it so you cannot tweak it (/proc/net, /proc/sys/net etc).
    In the future, if you have to face SYN floods again, go for Xen VPS for instance, at least you could fight back

    My host recommend that I try:
    "Try with nginx as a reverse proxy and let us know how it works."
    Unless there were some major changes to the TCP protocole last night and no one informed me, before a packet can reach the HTTP server backlog there must be a 3-way handshake sequence. During a SYN flood you are stuck (and will remain stuck) in the middle of it. It is a kernel problem, not an Apache/Nginx problem. The only help you could get is from your hosting company but... it looks like it's going to be tough

    I think that the number next to the IP (in my first post) is the number of processes that that IP is using on my VPS.
    If they are in the SYN_RECV state, they are just half-opened connections, no processes. They are just filling up your kernel backlog and when it's full, your server drops any further packet.

    What about the 195 ESTABLISHED connections ? Is that the average/normal traffic on your server ?
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

Similar Threads

  1. DDOS attack
    By habibjr in forum Dedicated Server
    Replies: 7
    Last Post: 09-03-2008, 09:24 AM
  2. ddos attack, been down for 2 days... HELP!
    By Qpad in forum Hosting Security and Technology
    Replies: 4
    Last Post: 05-24-2008, 01:55 PM
  3. DDOS attack
    By Hserver in forum Hosting Security and Technology
    Replies: 5
    Last Post: 10-06-2007, 03:30 AM
  4. DDOS attack
    By tax in forum Hosting Security and Technology
    Replies: 2
    Last Post: 04-22-2005, 07:56 PM
  5. DDOS Attack??
    By Asco in forum Hosting Security and Technology
    Replies: 24
    Last Post: 11-27-2004, 05:09 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •