Results 1 to 11 of 11
  1. #1
    Join Date
    Sep 2002
    Posts
    265

    under SYS Flood attack

    I am under sys flood attack, the ISP already installed mod_evasive & enabled tcp_syncookies, as well as IPtables/APF, it seems can not help much.

    My ISP also may offer CISCO PIX 501 at extra fee. however, I checked some posts at WHT, it seems such a hardware firewall can not do much at my case.

    So, I want to know what else I may do besides null routing my IP? thanks.

  2. #2
    Join Date
    Aug 2002
    Location
    Seattle
    Posts
    5,512
    As i've preached before, those types of modifications along with an optimized kernel will give you roughly 10,000 PPS of mitigation to play with.

    With many attacks rolling in at between 10,000 and 1M PPS that's only going to solve some of your problems.

  3. #3
    Join Date
    Jun 2004
    Location
    Tampa Florida
    Posts
    428
    There is no way to stop a syn flood. Just mittigate it. and if the people attacking can ramp it up past a certain point... then you will still go down.

    We usualy will put an openBSD box infront of the effected server. PF and some other tools inOpenBSD give you the ability to mittigate it about as much as possible. That or a Cisco router running TCP Intercept.
    Rock solid hosting and dedicated servers since 1998!
    StabilityHosting Where stability and uptime are king!

  4. #4
    Quote Originally Posted by surfmanjoe
    I am under sys flood attack, the ISP already installed mod_evasive & enabled tcp_syncookies, as well as IPtables/APF, it seems can not help much.

    So, I want to know what else I may do besides null routing my IP? thanks.
    My initial question would be 'Are you absolutely sure this is a SYN flood?

    Many of the website attacks today are actually a lower bandwidth attack by botnets which attack the website by exploiting poor/slow php code and inefficient mySQL lookups that are performed even on non-verified users. This results in locking up the server by running it out of resources. For security reasons I won't go into how this is done.

    In any case, either true SYN or the new botnet attack, your machine needs to be optimized at the kernel level, not just at the apache level. Additionally you may want to purchase some higher network-level DDoS protection from your ISP.

    In the case of a botnet attack (non-syn flood) there are several options you can install at the server (or network) level to defeat the attack and stay online.

    A hardware firewall is unnecessary.

    Null Routing should NOT be an option, unless you are getting 1-2gbps floods and are either exceeding the pipe for your ISP or are using more bw than your ISP cares to absorb for the $$ you are paying.
    Last edited by DiverGuy; 12-21-2006 at 06:18 PM.
    John Blazek
    Auroralink Communications
    Shell & Web Hosting

  5. #5
    Join Date
    Sep 2002
    Posts
    265
    well, how can I make sure if it is SYN flood? I am not professional on this problem. I just was told by the technician at ISP. thanks.

  6. #6
    Quote Originally Posted by surfmanjoe
    well, how can I make sure if it is SYN flood? I am not professional on this problem. I just was told by the technician at ISP. thanks.
    Who suggested NULL route? You or your ISP? and Why?
    Has your ISP given you any bandwidth numbers or pps numbers?

    If they claim it is a SYN flood, ask them what volume of SYN packets you are receiving.

    What exactly is happening to your server?
    Can you still get in?
    How many processes are running?
    How many copies of apache?
    What are the results of a netstat -an command?
    Do you see 100's of SYN from the same ip?
    What do your apache logs show?

    Are you paying your ISP to handle this? If so, they should be able to do all these things and tell you exactly what is going on.
    John Blazek
    Auroralink Communications
    Shell & Web Hosting

  7. #7
    Join Date
    Sep 2002
    Posts
    265
    SYS_FLOOD was advised by the ISP tech.

    1) What exactly is happening to your server?

    A: Very very slow. when I check under TOP command:

    17:52:40 up 20:03, 1 user, load average: 76.29, 71.93, 81.29
    454 processes: 427 sleeping, 26 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 94.6% 0.0% 5.0% 0.0% 0.3% 0.0% 0.0%
    cpu00 94.0% 0.0% 4.7% 0.3% 0.7% 0.0% 0.0%
    cpu01 94.4% 0.0% 5.1% 0.0% 0.3% 0.0% 0.0%
    Mem: 2074388k av, 1979892k used, 94496k free, 0k shrd, 27160k buff
    1698284k active, 214800k inactive
    Swap: 2096440k av, 2740k used, 2093700k free 398096k cached

    2) Can you still get in?

    A: Yes, pretty much slow.

    3) How many processes are running?

    A: PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    25840 nobody 15 0 19296 10M 3932 S 5.9 0.5 0:10 1 httpd
    20692 nobody 16 0 19872 11M 3940 S 3.5 0.5 1:01 1 httpd
    25880 nobody 16 0 21116 12M 3820 S 3.5 0.6 0:06 1 httpd
    25914 nobody 15 0 19504 10M 3816 S 3.5 0.5 0:07 0 httpd
    26076 nobody 16 0 19588 10M 3812 S 3.5 0.5 0:05 1 httpd
    26124 nobody 16 0 22752 13M 3828 R 3.5 0.6 0:05 0 httpd
    20556 nobody 15 0 23196 14M 3940 S 3.1 0.7 1:35 0 httpd
    25829 nobody 16 0 22596 13M 3832 S 3.1 0.6 0:08 0 httpd
    26053 nobody 15 0 23144 14M 3836 S 3.1 0.7 0:05 0 httpd
    20658 nobody 15 0 21008 12M 3932 S 2.5 0.6 1:17 0 httpd
    25882 nobody 15 0 19160 10M 3820 S 2.4 0.5 0:08 0 httpd
    25842 nobody 15 0 19944 11M 3816 S 2.2 0.5 0:07 1 httpd
    26118 nobody 15 0 19692 10M 3852 S 2.0 0.5 0:04 1 httpd
    20656 nobody 17 0 23408 14M 3844 R 1.8 0.7 1:22 1 httpd
    25946 nobody 15 0 19448 10M 3824 S 1.8 0.5 0:04 0 httpd
    26005 nobody 15 0 19144 10M 3812 S 1.8 0.5 0:05 0 httpd
    26142 nobody 15 0 19432 10M 3820 S 1.8 0.5 0:04 0 httpd
    14574 nobody 15 0 24036 15M 4016 S 1.6 0.7 5:08 0 httpd
    20691 nobody 17 0 21228 12M 3944 R 1.6 0.6 1:18 1 httpd
    25887 nobody 15 0 19688 10M 3820 S 1.6 0.5 0:07 0 httpd
    26066 nobody 16 0 19268 10M 3820 S 1.6 0.5 0:04 1 httpd
    14575 nobody 15 0 22568 13M 3988 S 1.4 0.6 5:29 0 httpd
    14577 nobody 15 0 28520 19M 4020 S 1.4 0.9 5:25 1 httpd
    14579 nobody 15 0 34976 25M 4016 S 1.4 1.2 5:35 1 httpd
    25835 nobody 15 0 19632 10M 3820 S 1.4 0.5 0:11 1 httpd
    26006 nobody 15 0 19448 10M 3824 S 1.4 0.5 0:05 1 httpd
    26038 nobody 15 0 19524 10M 3808 S 1.4 0.5 0:04 1 httpd
    26145 nobody 15 0 19420 10M 3808 S 1.4 0.5 0:04 0 httpd
    26512 nobody 15 0 19400 10M 3808 S 1.4 0.5 0:01 0 httpd
    18665 nobody 15 0 23048 14M 3848 S 1.2 0.6 3:23 1 httpd
    20598 nobody 16 0 23808 14M 3956 S 1.2 0.7 1:30 1 httpd
    25830 nobody 15 0 22880 13M 3836 S 1.2 0.6 0:09 1 httpd
    25921 nobody 16 0 19704 10M 3820 S 1.2 0.5 0:05 0 httpd
    25947 nobody 15 0 19256 10M 3924 S 1.2 0.5 0:05 1 httpd
    26117 nobody 15 0 19696 10M 3820 S 1.2 0.5 0:05 0 httpd
    26125 nobody 15 0 19508 10M 3816 S 1.2 0.5 0:05 0 httpd
    26139 nobody 15 0 19156 10M 3816 S 1.2 0.5 0:05 1 httpd
    25825 nobody 16 0 20080 11M 3824 R 1.1 0.5 0:10 1 httpd
    25902 nobody 16 0 19912 11M 3812 S 1.1 0.5 0:08 0 httpd
    25943 nobody 16 0 19216 10M 3820 R 1.1 0.5 0:06 1 httpd
    26138 nobody 16 0 19428 10M 3808 R 1.1 0.5 0:05 0 httpd
    26144 nobody 16 0 19420 10M 3824 S 1.1 0.5 0:05 0 httpd
    14584 nobody 16 0 23416 14M 4040 R 0.9 0.7 5:31 1 httpd
    25868 nobody 15 0 22468 13M 3884 S 0.9 0.6 0:08 0 httpd
    26027 nobody 16 0 19092 10M 3820 R 0.9 0.5 0:04 0 httpd
    26126 nobody 16 0 19636 10M 3816 R 0.9 0.5 0:05 0 httpd
    26132 nobody 16 0 19684 10M 3816 R 0.9 0.5 0:04 0 httpd
    14580 nobody 16 0 26956 18M 4028 R 0.7 0.8 5:31 1 httpd
    25838 nobody 17 0 19980 11M 3828 R 0.7 0.5 0:09 1 httpd
    26122 nobody 15 0 19080 10M 3828 S 0.7 0.5 0:05 0 httpd
    26610 root 16 0 5556 1452 892 R 0.7 0.0 0:00 0 top

    4) How many copies of apache?

    A: see above

    5) What are the results of a netstat -an command?

    A: Huge IPs list out. most status are TIME_WAIT

    6) Do you see 100's of SYN from the same ip?

    A: YES

    7) What do your apache logs show?

    No idea for now


    thanks.

  8. #8
    Join Date
    Sep 2002
    Posts
    265
    I just got apache access_log file shows like these;


    127.0.0.1 - - [21/Dec/2006:17:20:52 -0500] "GET /whm-server-status HTTP/1.0" 200 72865
    127.0.0.1 - - [21/Dec/2006:17:24:56 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [21/Dec/2006:17:26:37 -0500] "GET /whm-server-status HTTP/1.0" 200 72128
    127.0.0.1 - - [21/Dec/2006:17:31:43 -0500] "GET /whm-server-status HTTP/1.0" 200 72027
    127.0.0.1 - - [21/Dec/2006:17:34:26 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [21/Dec/2006:17:35:41 -0500] "GET /whm-server-status HTTP/1.0" 200 71048
    127.0.0.1 - - [21/Dec/2006:17:40:01 -0500] "GET /whm-server-status HTTP/1.0" 200 71546
    127.0.0.1 - - [21/Dec/2006:17:44:15 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [21/Dec/2006:17:45:20 -0500] "GET /whm-server-status HTTP/1.0" 200 73325
    127.0.0.1 - - [21/Dec/2006:17:52:03 -0500] "GET /whm-server-status HTTP/1.0" 200 73989
    127.0.0.1 - - [21/Dec/2006:17:54:17 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [21/Dec/2006:17:55:39 -0500] "GET /whm-server-status HTTP/1.0" 200 73696
    127.0.0.1 - - [21/Dec/2006:18:01:41 -0500] "GET /whm-server-status HTTP/1.0" 200 74087
    127.0.0.1 - - [21/Dec/2006:18:02:55 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [21/Dec/2006:18:05:46 -0500] "GET /whm-server-status HTTP/1.0" 200 73884
    127.0.0.1 - - [21/Dec/2006:18:10:18 -0500] "GET /whm-server-status HTTP/1.0" 200 74120

    any idea on my problem? thanks.

  9. #9
    You appear to have overlooked several of my questions..:

    Who suggested NULL route? You or your ISP? and Why?
    Has your ISP given you any bandwidth numbers or pps numbers?

    If they claim it is a SYN flood, ask them what volume of SYN packets you are receiving.
    These details would be very helpful in the 'remote diagnosis' you are asking for.

    Are you paying your ISP to handle this? If so, they should be able to do all these things and tell you exactly what is going on.
    If so, then they should be able to do all these things for you. After all that why you are paying them. right?

    ******
    100's of netstat entries 'from' the same IP would tend indentify a SYN flood, though that is not guaranteed. I'll take your word that you are looking at the 'source' and not the 'destination' IP.

    If it is a true (and harsh) SYN attack, then you would virtually be prevented from logging into an unprotected server due to 100% usage of resources.

    On the other hand, 94% CPU utilization points to a CPU resource attack, not a true/pure syn attack. So you may have a combination of both, or a poorly run or crude botnet attack.

    You'll need to adjust the syn and wait timeouts in your kernel using sysctl (these will vary between operating systems)

    You'll also need to optimize the apache settings for timeouts, and keep-alive connections.

    Depending on your apache logs, you may want to have your ISP install and properly configure mod_security.

    You'll want to modify iptables to rate limit connections from each IP.

    etc etc..

    Also, start collecting the IP's involved.. if they are not in the 1000's then you can add them to your firewall and just ignore SYN from those IPs.
    Last edited by DiverGuy; 12-21-2006 at 07:28 PM.
    John Blazek
    Auroralink Communications
    Shell & Web Hosting

  10. #10
    Join Date
    Sep 2002
    Posts
    265
    which command may list each IP's connection number?

    I tried

    netstat -apn|grep :80 |awk '{print $5}'|sort

    it lists all IPs, but I want to know the total connections number for each IP. thanks.

  11. #11
    Join Date
    May 2001
    Location
    Dayton, Ohio
    Posts
    4,962
    This is horribly written but this might help you:

    Code:
    netstat -natu | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -n
    -Mat Sumpter
    Director, Product Engagement
    Penton Media

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •