Preface:

After reading a few posts here about being hacked I elected to compose this short guide.

I won't profess to be a security expert by any means. While I'm sure industry collegues might be able to add to this I hope to offer a good starter in the hopes that other admins don't suffer losses like I did in my early days a few years ago.

My familiarity with Nix based systems is limited to the RedHat line thus my instructions will work on such systems but I can't count that they will on other Nix flavours. (although they should for the most part.)

This is directed to system administrators, not website administrators. If you are a website administrator and you feel you or your server has been hacked I urge you to backup all that you can from your site and contact your system/network administrator with what ever evidence you have to support your concerns.

Starters:

You will need SSH and Root access to your server to complete this. Have note pad open and prepare to copy and paste entries which point to the illegal activity for later reporting. Coffee and Valum are optional.

Suspicions, have I been hacked?

Check your bandwidth usage.

As a system administrator you should have a general idea of what your using for bandwidth. Tools such as MRTG are excellent to show your bandwidth trends. Excessive usage indicates that someone has comprimised your server to distribute warez or porn which is a very common purpose for hackers.

Check your disk usage:

[root@yourserver root]# df

Your output should be similar to:

Filesystem 1k-blocks Used Available Use% Mounted on
/dev/hda3 38376872 3461780 32965608 10% /
/dev/hdb1 19750244 3937284 14809704 22% /archive
/dev/hda1 101089 9043 86827 10% /boot
none 515612 0 515612 0% /dev/shm


We're looking for unusual usage here. As the admin you should have an idea of what your using.. If your numbers looks strange try to track down larger files and see whats up.

[root@yourserver root]# find '/' -size +50000k

This will seek out files on your server larger than 50 Megs. If thats normal adjust the number accordingly. Pirate CD's will be in the 500+ Meg range where as video files of quality are 50M - 1+ Gig in size.

Alternatly, you can look for files with common file extensions such is .bin, .rar, .iso etc....

[root@yourserver root]# find '/' -iname *.rar

This would find all files on your server with the .rar extension. Take note of the odd files and thier directories.

Run Chkrootkit

I scribbled a guide on installing and running chkrootkit here on WHT: http://www.webhostingtalk.com/showthread.php?s=&threadid=228109

Google any results you get for information/solutions on the particular trojan you've been infected with.

If so far you've gotten confirmation that thigns aren't right we need to press on.

Find out whos been logging in.

[root@yourserver root]# last | more

This will output what username from what location has logged into your server when.

Determine if your logs are running.

Hackers love to shut the loggers off to help cover thier tracks.

[root@yourserver root]# ps -aux | grep syslogd

You should see a line that looks like:

root 695 0.0 0.0 1460 536 ? S Jan05 0:35 syslogd -m 0 -a /


If you don't thats likely bad. Restart it:

[root@yourserver root]# /etc/rc.d/init.d/syslog restart

Find suspicious accounts/groups

We're looking for suspicious users. gamerz, pron, adm1n, test, qwerty and the like.. Anything that looks out of place. Before editing this file check with other servers or admins to see what thiers looks like. Some users such as shutdown, games, operator and the like are are normal upon install.

[root@yourserver root]# less /etc/passwd

The 1st column are your users. (tip "q" exits a less screen)

[root@yourserver root]# less /etc/shadow

Same deal, look for funky users.

[root@yourserver root]# less /etc/group

More of the same, looking for funky groups.

Ok, so if we've found and determined that we have some funkiness going on, it's time to get rid of the users. If you've seen interesting logins in (last | more) you would have found them in these files.

[root@yourserver root]# vi /etc/passwd

Scroll down to funky user and delete the entire line. Repeat this on all 3 of the files (passwd, shadow, groups) until all traces of un-authorized users are gone.

Shutting them down

We want to see if hacker has edit our crontab to run thier programs etc..

[root@yourserver root]# crontab -e

Should be empty for the most part. If you see something in here that doesn't belong, again check with other administrators/servers to see if in fact it belongs, if not - delete the line.

Now, check the history of the users to find out what he did to your box.

[root@yourserver root]# find '/' -iname .bash_history

If your hacker wasn't thorough they may have left you this gift. Edit the file in your favorite editor and copy the entire contents to your notepad. This is what the person was doing on your box. You can compare this to the information given in last | more to put the actions to the users IP/Hostname, this information should be submitted to the authorities to persue prosecution.

Now, lets go through and remove all of the files we no longer want on our sever. Using the notes taken during our file search remove the files.

At this point we've shut out the bad guys, and remove thier files from the system and also have notes and logs of who they were.. Retribution might be on the horizon. 1 Problem though; your system is still vulnerable. The hacker did afterall get in initially so that means they or others can again as well.. You can do a security lockdown (I'll write that up another night) but that possibly won't erase all traces.

We can't be certain at this point that we've really shut this person down no more then we can ensure that your server is 100% safe. Ultimatly, your best bet is to grab backups of your websites and re-install. Take a day or 2 hoping that our shut down here has scared the hacker away from your box but no more as you are open to the next hacker in line.

After a successful re-installation/migration/restore or whatever you do take the time to really secure your box. I will write what I know of doing this in another thread but you may want to consider hiring an industry professional to do this for you if your not comfortable with it.

I hope this is helpful to some, and I'm eager to see what other administrators can add to this write up so that I can learn more as well !

Best regards and safe serving.

Ryan

Nice howto. :)


Originally posted by MjrGaelic
[root@yourserver root]# less /etc/groups

^^ That should be "less /etc/group". Notice no "s" at the end.

<<< Signature removed >>>

Thanks for pointing that out.. No ammount of late night proof reading can save me from these hehe. I should start writing earlier in the day.

Request to admin was sent to correct.

Ryan

This guide wasn't perhaps inspired by:

http://forum.ev1servers.net/showthread.php?threadid=13172

at all, was it?

<<< Signature removed >>>

Actually no it wasn't but now that I reading MattDH's post on the above listed forum it's creepy the similarity which a) makes me feel good that I'm well on the right track b) makes me feel stupid for spending the time that I did researching this through various mailing lists found on google, and collegues in the industry.

For the most part I was actually inspired by http://www.security.uconn.edu/HackerCheckList.html#3
.

I am a casual Ensim How-To (I run Ensim Servers)reader but can honestly say that I haven't seen that particular post but wish I had =). Would have saved me a great deal of time

nice how to

<<< Signature removed >>>

MjrGaelic

good job regardless where it comes from and regardless useless comments

While (somewhat) useful, the above doesn't really tell you if you've been hacked.

Firstly, root DOES have crontab entries, especially if you have anything for a control panel and scripts that you need to update daily, nightly, hourly, whatever, so removing those entries could possibly be very dangerous. There's nothing wrong with a root cron entry.

Secondly, before doing anything else, you should check certain rpms against sources, verify that they're up to date and haven't been hacked. Otherwise you're seeing just what the hacker wants you to. It's unlikely that you'll run into a hacker that hasn't modified the ps, df, top and login binaries, so you'll need to reinstall those most often.

Thirdly, rootcheck is good, but it is only one tool. Most hackers detect it and work around it somehow. Use rpm to check out and verify that the binaries have not been modified (or whatever tool your system comes with.)

Logs can be modified, wtmp is easy enough to modify, nothing can be trusted at all if you even 'suspect' your server has been hacked. Once you've established that you have a uncompromised ps and lsof binary, you should use these to verify the processes you're running.

Double check everything!!

<<< Signature removed >>>

Its always best to start with a fresh drive and carefully copy stuff over

<<< Signature removed >>>

Originally posted by thelinuxguy
Its always best to start with a fresh drive and carefully copy stuff over

Oh, I agree there, but in most cases that's just not an option. It IS posssible to remove all traces of a rootkit from your server without formatting (done it a few times before actually), but you're always going to wonder if you still have some trace of the hack in your system.

Best response: format, reinstall.

<<< Signature removed >>>

Originally posted by wolfstream
Oh, I agree there, but in most cases that's just not an option. It IS posssible to remove all traces of a rootkit from your server without formatting (done it a few times before actually), but you're always going to wonder if you still have some trace of the hack in your system.

Best response: format, reinstall.


Yes i agree one slip up and it starts all over again

<<< Signature removed >>>

I've been looking for a How-to like this. Great job!

If you think you may have been rked, you can also run this:

http://www.rootkit.nl/projects/rootkit_hunter.html
Current release:
http://downloads.rootkit.nl/rkhunter-1.0.7.tar.gz
extrack, install, and rkhunter --checkall

<<< Signature removed >>>

Just out of curiosity, how does this compare to chkrootkit?
Nice tool, I'll probably implement that into my series of checks as well now (never can have too much verification going on yano ;))

<<< Signature removed >>>

RK Hunter works pratically the same as chkrootkit. I think it does only 1 less test or so.

It has much cleaner output though. :)

cmark you are way off, rkhunter finds alot of things chkrootkit fails to find. For example a t0rn infected box, everyeone that ran chkrootkit and went off that is still infected with some other items rkhunter has found

<<< Signature removed >>>