I have seen the MRTG graph and it is unbelieveable! For the hour 16 - 18, my server had about 4.4MBits outgoing transfer! Anyone think my server is being attack?

My server use name-based hosting with Analog and ProFTPd, so how can I check the individual log and know who is using up so much bandwidth?

Thanks,

Kevin

Here is my MRTG graph

try looking at tcpdump and looking for repeat connections from the same hosts. Also, install snort!

lol, ive been upto 1.2MB/s in an attack but thats just crazy, lol

Someone thought the machine was on a broadband connection, they were pretty surprised when they gave it all they had and it still kept responding, lol - the laughs I had that day...

thanks your comment, but can I "tcpdump" back to that 2 hours to see why there were 4MBits/s? How?

So do you think they are downloading from my server or what? I really confused why they can eat 4MBits/s :(

From my Analog, I can't see such a big difference in httpd traffic, but when I check out netstat sometime, it just say normally HTTP and MySQL traffic only. :(

Thanks,

Kevin

Is there any special software that goes through logs and detects attacks then fires back? Suppose you have a larger pipe and faster machine than he does =) Too bad you can't hit them back

Isn't green incoming? Which would mean 4.4 incoming.. Maybe you we're D(d)OSed?

green is outgoing to the internet. I have installed "portsentry" and it shows that the following log has been repeated so many times, thus, I think they wanna hack my server, but they failed.

Jan 2 01:10:28 server ipop3d[7720]: Login failure user=hunneyviet06 host=adsl-64-160-47-217.dsl.snfc21.pacbell.net
[64.160.47.217]
Jan 11 09:54:19 server ipop3d[17611]: Login failure user=account host=cl0168.pc.nus.edu.sg
[137.132.225.168]
Jan 11 09:52:27 server proftpd[16959]: server.227media.com (137.132.225.168[137.132.225.168])
- SECURITY VIOLATION: root login attempted.
Jan 2 16:07:13 server named[948]: lame server on '71.10.200.205.in-addr.arpa'
(in '10.200.205.in-addr.arpa'?): 205.200.16.65#53
Jan 4 23:12:08 server sendmail[17222]: g04IC8e17222: ruleset=check_rcpt,
arg1=<J812@MYDOMAIN.net>, relay=[202.101.18.130], reject=550 5.7.1 <J812@MYDOMAIN.net>...
Relaying denied. IP name lookup failed [202.101.18.130]

Did these fail attempts use up my bandwidth? How can I prevent them to attack my server again?

Thanks,

Kevin

This could be an ICMP attack. If you dont have an intrusion detection system in place, it wouldn't really show up in your logs, but it would take up large amounts of bandwidth in an attempt to DOS you. If that's what it is, it is actually quite easy for someone to launch with tons of bandwidth.

For example, if you wanted to attack Computer A, you send out ping's to <I>networks</I>B, C, and D, but you set the return IP Address to be Computer A. When those networks respond to what they think is a legitimate ping request, Computer A gets flooded with pings, and can sometimes crash, or at least slow down.

You should really have some sort of intrusion detection up and running, so you can determine this sort of thing. NTop and Snort are both good tools to use in this sort of scenario, and they're both free. You have to keep them running, though. They can't help you after the fact.

Please let me know if you'd like more information, or if my explination wasn't clear enough :)

-Dan

Robot Two, thanks your help, I understand what you mean, now, i've installed both portsentry and snort but it seems they have conflicts.

[**] [100:1:1] spp_portscan: PORTSCAN DETECTED from 1.2.3.4 (THRESHOLD 4 connections exceeded in 2 seconds) [**]
01/20-23:08:01.953258

[**] [100:2:1] spp_portscan: portscan status from 1.2.3.4: 5 connections across 5 hosts: TCP(0), UDP(5) [**]
01/20-23:08:05.234255


While 1.2.3.4 is one of my server IP

Also, the load average is really high after installing the snort:

load average: 7.23, 3.65, 2.44

I just need to run one of them is enough? Which one is better / can block the connection from an IP automatically when they detect that IP always attack the server?

btw, how can I disable "Ping" from other servers if not using the above packages?

Thanks,

Kevin

I'm gonna break this post up into a few different topics

-----------------------------------------------------------------------------
CONFLICT

Actually, that's not a conflict.

Snort is reporting that there is someone out there scanning your network, looking to see what types of servers you have running. This can be a prelude to an attack, or it can be nothing. A good analogy is walking down the street and seeing if any car doors are unlocked. You may or may not steal a car, but the fact that you are looking is suspicious.

There's also nearly nothing you can do about it, except to complain to whoever owns the IP address you are being scanned from (Snort should tell you this in the log entry).

I've been working with major ISP's for nearly 5 years, and it is pretty normal nowadays to get scanned once a day or more, without a followup attack -- so don't worry too much abou portscans. However, if an attack comes later on from the same address, you should link the two events.
----------------------------------------------------------------------------------

LOAD

Yes, your system load will jump significantly. This is because both snort and ntop will look at *every single packet* traveling accross the network, even if they are not destined for your machine (assuming that your machine can keep up with it).

This is both good and bad. It's good, because you gather lots and lots of information. It's bad because it bogs down your machine a bit.

A boon here is if you are running several machines. If you are, you can set it up so that you only need one machine to run Snort and NTop, and it can monitor your entire network. If you only have, say, one machine in a co-location, this option is not available to you.

----------------------------------------------------------------------------------

WHAT THEY DO (Do you need both?)

This is really up to you. It depends on what you want to do.

Snort will anylize network packets, and compare them to its ruleset of known hacking and denial-of-service attacks. If it finds a match, then an attack is likely in progress (note likely, not definately -- it can make mistakes), and it notes that in its log.

You can add or delete rules in the Snort ruleset. The more rules you have, the more resources snort will take.

Snort keeps a history of intrusion attempts. If you want to log attempted breakins, successful breakins, etc, Snort is the tool to do it.


NTop, on the otherhand, generates statistics about what network traffic looks like. How much of your bandwidth is http traffic vs ftp, UDP vs TCP, etc. It can view in realtime who has open connections to your machines, and from where. It can also tell you if your slowdown is because of ICMP traffic or something else.

You only need to use the toold you want. Ideally, you would have a machine dedicated to intrusion detection, and it would run both of these tools plus several more I wont mention here. However, the tradeoff for performance vs data is up to you. When it comes to securing your machine and your network, I recommend running as many of these sorts of utilities as you can afford to performance-wise.

-----------------------------------------------------------------------------------

Blocking ICMP

To defeat ICMP-based denial of service attacks, many server administrators simply dont allow their servers to respond to ICMP requests at all. Ideally, you want to do this with a firewall. I'm going to assume that you dont have this option available to you. I'm also going to assume that you're running a recent-ish version of linux.

This is the command to stop Linux from responding to ICMP alltogether:

.... continued:

echo "1" > /proc/sys/ipv4/icmp_echo_ignore_all

You can put that at the end of /etc/rc.d/rc.sysinit if you want it to defaul to that setting when the system reboots.

Of course, if you are not running a Linuc machine, the method to do this will be different depending on what it is (*BSD, Solaris, Irix, AIX, etc).

Again, let me know if you have any more questions, or if I haven't been clear enough in my explenation.

-Dan


------------------------------------------------------------------------------
Daniel Eisner
Network Security Specialist
http://www.3robots.com

quote:
--------------------------------------------------------------------------------
[**] [100:1:1] spp_portscan: PORTSCAN DETECTED from 1.2.3.4 (THRESHOLD 4 connections exceeded in 2 seconds) [**]
01/20-23:08:01.953258

[**] [100:2:1] spp_portscan: portscan status from 1.2.3.4: 5 connections across 5 hosts: TCP(0), UDP(5) [**]
01/20-23:08:05.234255

--------------------------------------------------------------------------------


Sorry, I didn't read that carefully enough.

The two most likely causes of this are either MRTG or a locally running DNS server. Both of these programs will connect to many machines on UDP, and might appear to Snort to be a portscan. If you have neither of these running, then it might be a cause for concern.

Dan, thanks for your help, it's much helpful in my situation.

Actually I only have 1 machine with RedHat 7.2 co-located, MRTG is on, my server is act as DNS servers as well and the performance is important, thus I think it's a better way to just switch off the ICMP services, right?

However, I can't find the path:

/proc/sys/ipv4/icmp_echo_ignore_all

in my server, so how can I block the ICMP?

Also, Snort cannot automatically block the services for the SPAM IP but just capture it in log files for me to check with?

And what "load avaerage" should be regarded as high? Above 1.00?

Thanks very much!

Kevin

Sorry, that should read "/proc/sys/net/ipv4/icmp_echo_ignore_all"

Also, you're right. Without 3rd party plugins or applications, snort simply watches what's going on, it doesn't take any action.

The answer is Yes, you can do all that stuff, but directions go beyond the scope of what I can cover in a forum. Check out snort's web site and read up on it. You'll also learn a bit about network security in the process.

Load average is sort of difficult to measure. generally, a load average above 1.0 means that the CPU is never idle. However, your server can still remain quick and responsive until the load gets much higher -- or it may not.

The only real way to tell is to connect to it and see how "snappy" it feels. If its too slow for you, kill snort.

Unfortunately, it sounds like any more advanced solutions for security and intrusion detection are going to be out of your price range (they are very expensive).

If you are interested, my company can perform a security audit on your server. If we did, you could probably proceed much more comfortably without running snort or ntop, given that you most likely do not want to pay the performance penalty. It is a bit expensive, though. If you are interested, send me an email at eisner@3robots.com, or check out our web site for more information.

Either way, best of luck.

-Dan

---------------------------------------------------------
Daniel Eisner
Network Security Specialist
http://www.3robots.com

Dan, yeah, it works now, all icmp request would be ignored and I kick the snort away, cause it really slow the server :(

Thanks again, Dan, you give me a clear picture on the server security matter! :)

Kevin

Just to let you know that i am developing a script that will monitor that bandwidth taken up by sites up to once a minute. I uses a primitive version on friday night when my server was doing 3mbs instead of the 300kbps, i had found the site and suspended it within 3 minutes.

great! can you share your script with us? :D i know many of us would need that!

Well its kind of being developed very very slowly. ive got 2 big projects to complete before i can even start on it.