Since 4 hrs ago, one of our server's http service down. We checked the server and found the following lines in the system log:

Dec 12 01:53:40 ss04 kernel: NET: 660 messages suppressed.
Dec 12 01:53:45 ss04 kernel: NET: 721 messages suppressed.
Dec 12 01:53:50 ss04 kernel: NET: 691 messages suppressed.
Dec 12 01:53:55 ss04 kernel: NET: 651 messages suppressed.
.....

and when we run netstat -n, it reported:

tcp 0 0 203.98.164.152:80 157.71.48.50:36104 SYN_RECV
tcp 0 0 203.98.164.152:80 159.235.73.72:338 SYN_RECV
tcp 0 0 203.98.164.152:80 159.69.209.65:22740 SYN_RECV
tcp 0 0 203.98.164.152:80 222.150.165.88:64791 SYN_RECV
tcp 0 0 203.98.164.152:80 219.192.51.33:56695 SYN_RECV
tcp 0 0 203.98.164.152:80 8.63.216.47:57220 SYN_RECV
tcp 0 0 203.98.164.152:80 83.1.129.42:43461 SYN_RECV
tcp 0 0 203.98.164.152:80 33.106.107.23:51050 SYN_RECV
tcp 0 0 203.98.164.152:80 169.244.51.101:22173 SYN_RECV
tcp 0 0 203.98.164.152:80 44.18.70.15:65124 SYN_RECV
tcp 0 0 203.98.164.152:80 162.8.104.37:5091 SYN_RECV
tcp 0 0 203.98.164.152:80 60.74.233.60:10398 SYN_RECV
tcp 0 0 203.98.164.152:80 12.68.108.10:53682 SYN_RECV
tcp 0 0 203.98.164.152:80 202.98.106.23:50549 SYN_RECV
tcp 0 0 203.98.164.152:80 12.207.145.73:31048 SYN_RECV
tcp 0 0 203.98.164.152:80 85.32.105.72:37464 SYN_RECV
tcp 0 0 203.98.164.152:80 47.82.53.39:5558 SYN_RECV
tcp 0 0 203.98.164.152:80 24.174.157.15:61778 SYN_RECV
...... (up to 1500 lines like this)

When we only allow a few IPs to access the server port 80, all websites work normal. Seems the problem doesn't come from the server itself.

Server details:
P4 2.8G CPU, 2GB RAM, Apache 1.3.31, directadmin, CentOS 3.3 with latest update.

We appreciate for any recommendations, thanks

Yes, is sync flood

apache should be still up but the max client has reached so it doesn't serve any pages ~

Dear Yupapa,

The websites still down but the number of apache client is only 45 right now. Can we add some rules in iptables solve it?

thx

I guess you can't stop the attack unless you can stop his upstream ~

And it is all spoofing ips which means all the ips are fake ~

Most likely you'll have to change all the sites ip to a new ip and remove the Ddos ip from the server ~ :(

You may want to try to remove 203.98.164.152 from apache so that it doesn't listen to it. Apache then should work fine ~

We'll do this if there is no any other solution. thanks

problem solved :)

add the following line in /etc/sysctl.conf and reboot the server:

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

Oh really ?!? :eek4: ~~ Good to hear

Last time I had the same problem and I tried everything...
did an echo 1 > /proc/sys/net/ipv4/tcp_syncookies

but no luck... so I end up moving everyone to new ips ~ :(

yes, it works after we add the line and reboot the box. may be your kernel didn't support the tcp_syncookies option.

Originally posted by Tiga
yes, it works after we add the line and reboot the box. may be your kernel didn't support the tcp_syncookies option.

How do you understand whether the kernel supports the tcp_syncookies option or not?

In your config file grep for CONFIG_SYN_COOKIES

If it says:
# CONFIG_SYN_COOKIES is not set
you don't have it enabled

If it says
CONFIG_SYN_COOKIES=y

You still have to enable it in other places though...

I have same problem.
Kernel config has CONFIG_SYN_COOKIES=y and /etc/sysctl.conf has net.ipv4.tcp_syncookies = 1

i also have apache with mod_dosevasive and apf install.

What should I do?

-asc-

Originally posted by asc2000
I have same problem.
Kernel config has CONFIG_SYN_COOKIES=y and /etc/sysctl.conf has net.ipv4.tcp_syncookies = 1

i also have apache with mod_dosevasive and apf install.

What should I do?

-asc-

You also see a lot of SYN_RECV lines in netstat? Your sysctl.conf indicates that syn cookies are enabled on network restart, but you may not have them enabled now.

What's the output of:


sysctl -a |grep cookie


If you have syn cookies enabled and you still see a lot of syns in netstat, it's likely that they're not from spoofed IP addresses and you actually didn't have the same problem.

Additionally, the lines like "Dec 12 01:53:40 ss04 kernel: NET: 660 messages suppressed" in your logs really don't say much unless you also post which lines were suppressed.

Originally posted by bitserve
You also see a lot of SYN_RECV lines in netstat? Your sysctl.conf indicates that syn cookies are enabled on network restart, but you may not have them enabled now.

What's the output of:


sysctl -a |grep cookie




I have only one SYN_RECV
output of sysctl -a |grep cookie is net.ipv4.tcp_syncookies = 0

Edit:
problem solved after setting SYSCTL_SYNCOOKIES on apf config.

-asc-