Web Hosting Talk


Go Back   Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Detect and Clean your hacked server!! T0rnkit v8
Reply

Hosting Security and Technology Configuring and optimizing web hosting servers and operating systems, developing administration scripts, building servers, protecting against hackers, and general security (SSL certificates, etc.)

 
Thread Tools Search this Thread Display Modes
  #1  
Old 03-13-2004, 02:24 AM
admin0 admin0 is offline
View Beta Profile
Web Hosting Master
 
Join Date: Dec 2001
Location: Singapore
Posts: 644
Detect and Clean your hacked server!! T0rnkit v8

Hi,

Around 12 [rooted] servers were passed to me in these 24 hours, all due to the latest CPanel/WHM bug, and after inspection, all servers had the same rootkit in common. [Tornkit v8]

I am including this so that you all diag and clean up your hacked server.

First of all,
Login to WHM as root
Click Tweak Settings
and please remove the tick from
[] Allow cPanel users to reset their password via email


1.
run chkrootkit, and you will see some INFECTED lines. It will also report that some process are hidden from the ps

chkrootkit

Checking `ifconfig'... INFECTED
Checking `login'... INFECTED
Checking `pstree'... INFECTED
and also:
Checking `lkm'... You have X process hidden for ps command
Warning: Possible LKM Trojan installed


2.
/etc/init.d/syslog restart

Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [FAILED]
Starting kernel logger: [ OK ]

3.
top

top: error while loading shared libraries: libncurses.so.4: cannot open shared object file: No such file or directory

4
tail /etc/rc.d/rc.sysinit

# Xntps (NTPv3 daemon) startup..
/usr/sbin/xntps -q

--------------------------------------------------------


OK.. looks like someone got to your server as well. Since we know what rootkit it is, let us investigate further.

Configuration files
<please use cat /path/filename/ to read what the files contain>


/usr/include/file.h (for file hiding)
/usr/include/proc.h (for ps proc hiding)
/lib/lidps1.so (for pstree hiding)
/usr/include/hosts.h (for netstat and net-hiding)
/usr/include/log.h (for log hiding)
/lib/lblip.tk/ (backdoored ssh configuration files are in this directory)
/dev/sdr0 (systems md5 checksum)
/lib/ldd.so {placing tks(sniffer), tkp(parser) and tksb(log cleaner)}


Infected Binaries:


top, ps, pstree lsof, md5sum, dir, login, encrypt,ifconfig,find,ls,slocate,
tks,tksb,top,tkpnetstat,pg,syslogd,sz

Infected Librairies:

libproc.a,libproc.so.2.0.6,libproc.so

BackDoor: (located at /lib/lblip.tk)

shdc
shhk.pub
shk
shrs


--------------------------------------------------------
Now, Lets start the cleaning process:


1.
pico /etc/rc.d/rc.sysinit

remove the lines that show

# Xntps (NTPv3 daemon) startup..
/usr/sbin/xntps -q


2.
reboot the system

WARNING: 2 servers got their kernel removed after reboot.
If your's is the case and that is what the DataCenter complains after reboot, please ask them to do the following:

reboot the system using the redhat CD into rescue mode
chroot to the /mnt/sysimage
reinstall kernel packages

that should fix it.

-- since already in resuce mode, perhaps also ask them to --force install the following rpm's

procps*.rpm
psmisc*.rpm
findutils*.rpm
fileutils*.rpm
util-linux*.rpm
net-tools*.rpm
textutils*.rpm
sysklogd*.rpm

3.
After the system is up

cd /lib
rm -rf lblip.tk


3.
remove the configuration files given above.

4.
cat /etc/redhat-release
note down your version of redhat, then from
www.rpmfind.net
search for the following rpm's

procps*.rpm
psmisc*.rpm
findutils*.rpm
fileutils*.rpm
util-linux*.rpm
net-tools*.rpm
textutils*.rpm
sysklogd*.rpm

-- and rpm --force install them


5.
if you see the hosts.h file, it says to hide all IP's from

cat /usr/include/hosts.h
193.60

thus, if you want, you can block all the IP's from 193.60 to your server via iptables.

6.
If all goes OK,
please reboot the server, and run chkrootkit again...

You should be OK!

Cheers!

__________________
███ Tech Support/Solutions => www.admin0.com
███ VoIP Setup/Customizations => www.voipbro.com
███
███ fulltime sysadmin since 1997!


Last edited by admin0; 03-13-2004 at 02:38 AM.
Reply With Quote
Sponsored Links
  #2  
Old 03-13-2004, 02:49 AM
Steven Steven is offline
View Beta Profile
Temporarily Suspended
 
Join Date: Mar 2003
Location: California USA
Posts: 9,735
Nice post. Was going to make one up but guess you beat me to it =)

Reply With Quote
  #3  
Old 03-13-2004, 02:52 AM
6PS-Chris 6PS-Chris is offline
View Beta Profile
Junior Guru
 
Join Date: Sep 2002
Location: Nashville, TN
Posts: 230
EXCELLENT INFORMATION!! A++

Too bad that wasn't around about 16 hours earlier. I was able to figure all that out on my own, I'm proud.

-ee99ee

__________________
ServerMotion

Reply With Quote
Sponsored Links
  #4  
Old 03-13-2004, 03:09 AM
Winkie Winkie is offline
View Beta Profile
Web Hosting Master
 
Join Date: Aug 2002
Location: UK
Posts: 846
If your server is infected, flatten and reinstall, by not reinstalling you are putting your data, your computer, your clients data and their computers at risk.

__________________
my $Winkie = skills('Technical Advisor', 'Network Technician', 'Perl Programmer', 'Server Admin');
I'm currently looking for employment as all of the above.

Reply With Quote
  #5  
Old 03-13-2004, 03:14 AM
Steven Steven is offline
View Beta Profile
Temporarily Suspended
 
Join Date: Mar 2003
Location: California USA
Posts: 9,735
Winkie reinstall is not always possible

Reply With Quote
  #6  
Old 03-13-2004, 03:16 AM
Winkie Winkie is offline
View Beta Profile
Web Hosting Master
 
Join Date: Aug 2002
Location: UK
Posts: 846
Quote:
Originally posted by thelinuxguy
Winkie reinstall is not always possible
Yes it is, it's also the only choice.

__________________
my $Winkie = skills('Technical Advisor', 'Network Technician', 'Perl Programmer', 'Server Admin');
I'm currently looking for employment as all of the above.

Reply With Quote
  #7  
Old 03-13-2004, 03:44 AM
admin0 admin0 is offline
View Beta Profile
Web Hosting Master
 
Join Date: Dec 2001
Location: Singapore
Posts: 644
well,

basically, in the short run, reinstall is NOT always possible, and there is need for quick patches and removal, so that we are able to get backup of the server, ensure that there are no more rootkits active, and that the potential exploits via which rootkits can be installed are all closed.

after all this is done, a time can be fixed with the DC, and after notifying the clients, a reinstall can be done, but again this depends on if it was a script-kiddie who just managed to install a rootkit, leaving the README there, or if indeed there is a potential threat.

Finally, in my case, it depends on the wish of my clients if they would like to reinstall or be happy after it is cleaned



__________________
███ Tech Support/Solutions => www.admin0.com
███ VoIP Setup/Customizations => www.voipbro.com
███
███ fulltime sysadmin since 1997!

Reply With Quote
  #8  
Old 03-13-2004, 04:29 AM
Winkie Winkie is offline
View Beta Profile
Web Hosting Master
 
Join Date: Aug 2002
Location: UK
Posts: 846
^^ I agree it might be technically difficult in the short term, but you should always have a plan for additional servers to balance the load or similar, but at least you have the right idea

It's impossible to check every file on your system for an exploit, and incredibly easy to leave one lying around.

__________________
my $Winkie = skills('Technical Advisor', 'Network Technician', 'Perl Programmer', 'Server Admin');
I'm currently looking for employment as all of the above.

Reply With Quote
  #9  
Old 03-13-2004, 06:57 AM
MattF MattF is offline
View Beta Profile
Web Hosting Master
 
Join Date: Nov 2000
Location: UK
Posts: 1,962
Make sure you check for setuid binarys too after cleaning up. A clever [insert word] might place a binary with root setuid in userspace and execute it at a later date.

__________________
MattF

Reply With Quote
  #10  
Old 03-13-2004, 10:31 AM
cem cem is offline
View Beta Profile
Junior Guru Wannabe
 
Join Date: Feb 2004
Posts: 46
I hope i wont get banned or something for posting these links but here it goes:


http://lists.debian.org/debian-user/.../msg00788.html

http://66.102.11.104/search?q=cache:...hl=en&ie=UTF-8


These sites gave me all the info i needed to remove tornkit8 - good luck!

Reply With Quote
  #11  
Old 03-13-2004, 12:13 PM
igorkl igorkl is offline
View Beta Profile
Junior Guru Wannabe
 
Join Date: Feb 2004
Location: Delray Beach, FL
Posts: 65
Hi!
Thanks for the really useful post. I got a question though:

Where else I can download that packages? Red Hat Network is down now till 8 pm EST, and rpmfind.net wont load, could be overloaded. I got Red Hat Enterprise Linux ES release 3 (Taroon Update 1).

BTW my syslogd wont start, phone is beeping every 5 min.

Reply With Quote
  #12  
Old 03-14-2004, 02:32 AM
elmister elmister is offline
View Beta Profile
Web Hosting Master
 
Join Date: Mar 2004
Posts: 565
You should also reinstall the following packages, they were trojanized in 3 boxes i cleaned

coreutils
perl (yes, use the trojan detect feature in WHM, some perl files were trojanized!)
lsof
slocate
VERY IMPORTANT, SSH is trojanized
openssh-clients
openssh-server


For redhat 9, there are a lot of ftp mirrors you can use to get the packages, for RHEL, it's closed, if you are in a hurry, send me a pm, i'll provide you with the packages... but you shouldn't trust anybody's else (than redhat) packages (i cleaned the first 2 boxes using up2date packages, and had to copy them to the third one)


Last edited by elmister; 03-14-2004 at 02:37 AM.
Reply With Quote
  #13  
Old 03-14-2004, 06:54 AM
WWWhost WWWhost is offline
View Beta Profile
Web Hosting Master
 
Join Date: Dec 2000
Posts: 951
hi,
my servers are running WHM 8.5.4 cPanel 8.5.4-R72
witouht password reset feature. Is there a security risk?
I don't think so. Please correct me if wrong.

Thanks a lot

Reply With Quote
  #14  
Old 03-14-2004, 08:28 AM
2webhost 2webhost is offline
View Beta Profile
Disabled
 
Join Date: Nov 2003
Posts: 82
Checking `bindshell'... INFECTED (PORTS: 465)

Reply With Quote
  #15  
Old 03-14-2004, 12:00 PM
Ramprage Ramprage is offline
View Beta Profile
Web Ninja
 
Join Date: May 2002
Location: Canada
Posts: 1,266
Quote:
Originally posted by 2webhost
Checking `bindshell'... INFECTED (PORTS: 465)
This is normal on Cpanel systems. If you see LOGIN or PSTREE infected then you have t0rnkit

__________________
Upload Guardian 2.0 • Sign up for our early beta
• Server wide security protection for your server

Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement: