hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Need help, DDoS attack on my VPS.. down for 2 days now
Reply

Forum Jump

Need help, DDoS attack on my VPS.. down for 2 days now

Reply Post New Thread In Hosting Security and Technology Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old
Junior Guru Wannabe
 
Join Date: Aug 2008
Posts: 35

Need help, DDoS attack on my VPS.. down for 2 days now


Hey guys, I'm in need of some help. For the past two days there has been a DDoS attack on my VPS.

My VPS specs are fairly small, so it is easy to take down, with only 512mb RAM, and a 666MHz CPU. I'm running the latest CentOS.

I've used the command:
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | sed -e s/'::ffff:'/''/g|cut -d: -f1 | sort | uniq -c | sort -n

To check the IP's that are hitting me, and there seems to be hundreds of different ones from all different countries.

" 26 99.147.202.54
28 142.59.192.124
29 119.160.178.45
29 67.206.209.63
36 202.27.218.72
47 66.75.49.158
67 87.3.160.142
130 80.117.212.205"

" 30 70.153.64.140
30 85.94.123.78
38 82.249.18.116
39 66.75.49.158
51 190.213.16.4
51 80.54.48.5
79 87.11.54.124
116 87.3.160.142"

" 36 121.96.116.63
38 85.94.123.78
46 70.153.64.140
59 80.54.48.5
69 190.213.16.4
139 87.11.54.124"

" 44 81.164.95.51
52 190.213.16.4
57 76.3.94.140
109 219.93.18.98
186 82.117.202.46
187 189.127.141.70
208 212.156.145.206"

And the list never ends.. Every few minutes all the IP's change.


As you can see the IP's that are using up all my resources are random, and change about once every minute. I've tried adding the most resource consuming IP's to my iptables, with no luck - as more and more IP's will pop up with 200+ processes in use.

I've got (D)Dos deflate installed, and from what I can see it doesn't seem to be working..


I'm stuck here, what could I possibly do to get my site back online, with limited money resources? My host recommend that I try:

"Try with nginx as a reverse proxy and let us know how it works."

What is this, and how would I use this?

Any help at all would be highly appreciated,
Matt.



Sponsored Links
  #2  
Old
Aspiring Evangelist
 
Join Date: Mar 2009
Location: /home/khunj
Posts: 398
Can you try those commands, at least it will give an idea of what kind of flood it is :

Code:
# netstat -nt | grep ':80 ' | awk '{print $6}' | sort | uniq -c
# netstat -s
# dmesg | tail -n 20
And what kind of VPS : Xen, OpenVZ... ?

__________________
NinTechNet
★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
★ NinjaMonitoring : Monitor your website for suspicious activities.

  #3  
Old
Junior Guru Wannabe
 
Join Date: Aug 2008
Posts: 35
Here are the results:

Code:
[root@server ~]# netstat -nt | grep ':80 ' | awk '{print $6}' | sort | uniq -c
     20 CLOSE_WAIT
    195 ESTABLISHED
     33 FIN_WAIT1
     73 FIN_WAIT2
      1 LAST_ACK
    745 SYN_RECV
     13 TIME_WAIT
Code:
[root@server ~]# netstat -s
Ip:
    9256571 total packets received
    0 forwarded
    0 incoming packets discarded
    7732206 incoming packets delivered
    2343368 requests sent out
    881 dropped because of missing route
Icmp:
    1255 ICMP messages received
    704 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 1250
        timeout in transit: 5
    8 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 8
IcmpMsg:
        InType3: 1250
        InType11: 5
        OutType3: 8
Tcp:
    196 active connections openings
    38248 passive connection openings
    8081 failed connection attempts
    17107 connection resets received
    194 connections established
    7730325 segments received
    2313529 segments send out
    30366 segments retransmited
    360 bad segments received.
    148401 resets sent
Udp:
    2051 packets received
    8 packets to unknown port received.
    0 packet receive errors
    2051 packets sent
TcpExt:
    8081 resets received for embryonic SYN_RECV sockets
    153089 packets pruned from receive queue because of socket buffer overrun
    25193 packets dropped from out-of-order queue because of socket buffer overrun
    4 packets rejects in established connections because of timestamp
    63886 delayed acks sent
    15 delayed acks further delayed because of locked socket
    Quick ack mode was activated 9627 times
    503145 times the listen queue of a socket overflowed
    503146 SYNs to LISTEN sockets ignored
    65 packets directly queued to recvmsg prequeue.
    1448 packets directly received from backlog
    14583 packets directly received from prequeue
    330668 packets header predicted
    14 packets header predicted and directly queued to user
    581661 acknowledgments not containing data received
    166671 predicted acknowledgments
    14 times recovered from packet loss due to fast retransmit
    3323 times recovered from packet loss due to SACK data
    Detected reordering 35 times using FACK
    Detected reordering 32 times using SACK
    Detected reordering 17 times using time stamp
    41 congestion windows fully recovered
    53 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 189
    204 congestion windows recovered after partial ack
    1365 TCP data loss events
    TCPLostRetransmit: 2
    892 timeouts after SACK recovery
    268 timeouts in loss state
    4842 fast retransmits
    362 forward retransmits
    2648 retransmits in slow start
    6850 other TCP timeouts
    434 sack retransmits failed
    256042 packets collapsed in receive queue due to low socket buffer
    21335 DSACKs sent for old packets
    576 DSACKs sent for out of order packets
    2876 DSACKs received
    1 DSACKs for out of order packets received
    351 connections reset due to unexpected data
    12704 connections reset due to early user close
    483 connections aborted due to memory pressure
    1122 connections aborted due to timeout
    18 times unabled to send RST due to no memory
IpExt:
Code:
[root@server ~]# dmesg | tail -n 20
ip_conntrack: CT 1521102616: table full, dropping packet.

It is OpenVZ.


Last edited by Phatmat; 10-04-2009 at 10:29 PM.
Sponsored Links
  #4  
Old
Web Hosting Master
 
Join Date: Nov 2007
Location: India
Posts: 843
nginx is webserver it is best to install for this environment,but better to know to stop this issue

__________________
HostNotch Hosting Services 99.9% uptime Shared Hosting, Reseller Hosting
yajur | Sales Team
CPanel Hosting R1 Soft Offsite-Backup Great Uptime
http://hostnotch.com sales @ hostnotch.com

  #5  
Old
Temporarily Suspended
 
Join Date: Oct 2009
Posts: 58
Apparently your ip_conntrack table is full, you can review your table
with:

# cat /proc/net/ip_conntrack

The max number of connections is set in

# cat /proc/sys/net/ipv4/ip_conntrack_max

You can increase it with:

# echo "some number" > /proc/sys/net/ipv4/ip_conntrack_max

Which hopefully will help. looks SYN related.

  #6  
Old
Junior Guru Wannabe
 
Join Date: Aug 2008
Posts: 35
Quote:
Originally Posted by Moonster View Post
Apparently your ip_conntrack table is full, you can review your table
with:

# cat /proc/net/ip_conntrack

The max number of connections is set in

# cat /proc/sys/net/ipv4/ip_conntrack_max

You can increase it with:

# echo "some number" > /proc/sys/net/ipv4/ip_conntrack_max

Which hopefully will help. looks SYN related.
What will this do? Allow more connections to the VPS?

I'm already out of RAM, so wouldn't this cause the DDoS attack to create more process, this make things worse?

I'm wondering, is there some way to prevent an IP having more than one process running? Some of the IP's (listed in my first post) have 200+ processes running, could I put a limit of say 10 here?

Let me know if my terminology is wrong, as from my current knowledge I think that the number next to the IP (in my first post) is the number of processes that that IP is using on my VPS.

- Matt.

  #7  
Old
Aspiring Evangelist
 
Join Date: Mar 2009
Location: /home/khunj
Posts: 398
A small SYN flood.
It's a bit weird your VPS cannot stand 700 half-opened connection.
Unfortunately, there's nothing you can do at the server level because you are using OpenVZ. It uses a single kernel, every users share it so you cannot tweak it (/proc/net, /proc/sys/net etc).
In the future, if you have to face SYN floods again, go for Xen VPS for instance, at least you could fight back

Quote:
My host recommend that I try:
"Try with nginx as a reverse proxy and let us know how it works."
Unless there were some major changes to the TCP protocole last night and no one informed me, before a packet can reach the HTTP server backlog there must be a 3-way handshake sequence. During a SYN flood you are stuck (and will remain stuck) in the middle of it. It is a kernel problem, not an Apache/Nginx problem. The only help you could get is from your hosting company but... it looks like it's going to be tough

Quote:
I think that the number next to the IP (in my first post) is the number of processes that that IP is using on my VPS.
If they are in the SYN_RECV state, they are just half-opened connections, no processes. They are just filling up your kernel backlog and when it's full, your server drops any further packet.

What about the 195 ESTABLISHED connections ? Is that the average/normal traffic on your server ?

__________________
NinTechNet
★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
★ NinjaMonitoring : Monitor your website for suspicious activities.

Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
DDOS attack habibjr Dedicated Server 7 09-03-2008 09:24 AM
ddos attack, been down for 2 days... HELP! Qpad Hosting Security and Technology 4 05-24-2008 01:55 PM
DDOS attack Hserver Hosting Security and Technology 5 10-06-2007 03:30 AM
DDOS attack tax Hosting Security and Technology 2 04-22-2005 07:56 PM
DDOS Attack?? Asco Hosting Security and Technology 24 11-27-2004 05:09 PM

Related posts from TheWhir.com
Title Type Date Posted
How Prepared are You for the Changing DDoS Landscape? Here’s 5 Must-Knows for Every Service Provider Webinars 2014-12-09 16:06:57
DDoS Attack Hits Australian ISP Cirrus Communications Web Hosting News 2014-08-01 12:29:18
DDoS Attack Targets 123-reg Customer, Impacts Shared Hosting Sites Web Hosting News 2014-05-01 08:33:46
.CN Domain Service Restored Following Massive DDoS Attack Web Hosting News 2013-08-26 17:03:15
Three DNS Hosting Providers Report Possibly Linked DDoS Attacks Web Hosting News 2013-06-05 16:50:15


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
WHT Host Brief Email:

We respect your privacy. We will never sell, rent, or give away your address to any outside party, ever.

Advertisement:
Web Hosting News:
WHT Membership
WHT Membership



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?