hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : [FEATURED] SSHD Rootkit Rolling around
Reply

Forum Jump

SSHD Rootkit Rolling around

Reply Post New Thread In Hosting Security and Technology Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old
Problem Solver
 
Join Date: Mar 2003
Location: California USA
Posts: 13,168

SSHD Rootkit Rolling around


Quick survey, anyone seen a rootkit being used to send spam through sshd involving a library called 'libkeyutils.so.1.9'?

If so what OS did you see it on?

__________________
Steven Ciaburri | Proactive Linux Server Management - Rack911.com
System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
Managed Servers (AS62710), Server Management, and Security Auditing.
www.HostingSecList.com - Security notices for the hosting community.


Thread Summary
Thread Summary UPDATE (Feb 21): Adding fire to the local vulnerability theory, cPanel has just released the following statement. cPanel is not the cause of every rooted server obviously, but merely one of the avenues through which server credentials were stolen.

Quote:
You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.

As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.
UPDATE (Feb 21): Several Linux anti-malware scanners such as AVG now detect the malicious libkeyutils files based on signature instead of just name.

UPDATE (Feb 20): Evidence is increasingly pointing towards a local vulnerability. The exploit filename also appears to be changing: libkeyutils-1.2.so.2 is popping up on CentOS 5.

--

If /lib64/libkeyutils.so.1.9 or /lib/libkeyutils.so.1.9 exist on your server, it is very likely that your server has been compromised at the root level and is currently sending out spam. Removing this file may be a temporary fix, but since the attack vector is still unknown, that is not likely a permanent fix. At this point, if your server has been rooted, the only 100% way to clean your server is to wipe your drives and do a clean installation.

Possibilities being discussed in this thread include a 0-day exploit of SSHD itself, curl vulnerabilities or even a local vulnerability attacking users through software like Adobe Flash and gaining root access to their servers via their computers.

Based on community input, it appears that both RHEL-based and Debian servers are affected. Servers with control panels such as cPanel, DirectAdmin, and Plesk are also affected. Servers with both standard and non-standard SSH ports are vulnerable and even servers that only accept key authentication have been compromised. Consider all passwords (including root) and private/public keys compromised. If you've made SSH connections to other servers from your exploited server, that login information is likely also compromised.

Recommended Actions: Since we still do not know the attack vector, we can only provide guidelines for things you should probably do.
  • Change all of your root passwords and key pairs from a clean computer
  • Keep your server software up-to-date
  • Disable root logins and/or firewall off your SSH port
  • Upgrade Flash and Java on your computers
  • Do malware scans on your computers
  • Keep checking this thread for updates! This thread summary will be constantly updated when we have new information.

WARNING: There are multiple scripts floating around the internet that promise to automatically clean up your server, but please be aware that they are not guaranteed to fix anything and have the potential to cause more problems. Run them at your own risk!

Contributors: Orien

Share This Summary:

Sponsored Links
  #2  
Old
Web Hosting Master
 
Join Date: Nov 2004
Location: Australia
Posts: 1,529
The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

CentOS release 6.3 (Final)
md5sum /lib64/libkeyutils.so.1.9
c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

-rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

rpm -qf /lib64/libkeyutils.so.1.9
file /lib64/libkeyutils.so.1.9 is not owned by any package

uname -r: 2.6.32-279.14.1.el6.x86_64.debug

  #3  
Old
Temporarily Suspended
 
Join Date: Mar 2012
Location: Tampa, FL =)
Posts: 1,748
I too can confirm this. Currently working with clients with spam issues and it is present. I checked other boxes we run and own and the library is no where to be found. It is only found on spam infested machines.

uname -a
2.6.32-042stab059.7

md5sum /lib64/libkeyutils.so.1.9
d81217186da61125f4dad7a87857b697 /lib64/libkeyutils.so.1.9

rpm -qf /lib64/libkeyutils.so.1.9
file /lib64/libkeyutils.so.1.9 is not owned by any package

Sponsored Links
  #4  
Old
Problem Solver
 
Join Date: Mar 2003
Location: California USA
Posts: 13,168
Quote:
Originally Posted by brianoz View Post
The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

CentOS release 6.3 (Final)
md5sum /lib64/libkeyutils.so.1.9
c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

-rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

rpm -qf /lib64/libkeyutils.so.1.9
file /lib64/libkeyutils.so.1.9 is not owned by any package

uname -r: 2.6.32-279.14.1.el6.x86_64.debug

They are not logging in with root, nor are they even spawning a bash process.
If the lib is moved out, and sshd is restarted they cannot login anymore fwiw.

The key is finding out how they are getting in. Fully upgraded, ssh key restricted sshd, on non-standard ports are being compromised.
None of my customers are, but I have been getting alot of sales inquiries with this issue so I don't know the full history of the machines.

Seeing it on centos 5, centos 6, cloudlinux 5, cloudlinux 6.

__________________
Steven Ciaburri | Proactive Linux Server Management - Rack911.com
System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
Managed Servers (AS62710), Server Management, and Security Auditing.
www.HostingSecList.com - Security notices for the hosting community.


Last edited by Steven; 02-08-2013 at 01:27 PM.
  #5  
Old
Aspiring Evangelist
 
Join Date: Sep 2000
Location: New Jersey
Posts: 380
I haven't seen this yet, but will keep my eye out.

64 bit only systems are what you are seeing? Are tcpwrappers or firewall offing ssh making a difference?

__________________
John Quaglieri - InterServer, Inc

  #6  
Old
Web Hosting Master
 
Join Date: Nov 2004
Location: Australia
Posts: 1,529
Firewalling off ssh stopped them on the machine I was looking at.
And the machine was 64 bit.

FWIW I suspect they are getting in initially some other way than ssh, but have no evidence.

  #7  
Old
Problem Solver
 
Join Date: Mar 2003
Location: California USA
Posts: 13,168
John,
Iptables will stop them.
Tcpwrappers does not.

Brianoz,
Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.

__________________
Steven Ciaburri | Proactive Linux Server Management - Rack911.com
System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
Managed Servers (AS62710), Server Management, and Security Auditing.
www.HostingSecList.com - Security notices for the hosting community.

  #8  
Old
Web Hosting Master
 
Join Date: Nov 2004
Location: Australia
Posts: 1,529
Quote:
Originally Posted by Steven View Post
Brianoz,
Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.
Steven - sorry for being unclear. I didn't mean to imply the initial attack was from ssh; what I meant was that an iptables block on ssh stopped them reconnecting, exactly as you've seen.

Is the following consistent with what you've seen?

1. User account compromised at PHP level
2. Compromised account used to hack root and backdoor sshd via libkeyutils
3. Spam sent

The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.

  #9  
Old
Newbie
 
Join Date: Feb 2006
Posts: 19
A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/". Would anyone of you be able to dump a copy of the file(s) at sourceforge +.net/tracker/ +?group_id=155034&atid=794187 or send it to me for analysis? Much appreciated, TIA.


Last edited by unSpawn; 02-09-2013 at 09:28 AM. Reason: //More *is* more
  #10  
Old
Newbie
 
Join Date: Feb 2006
Posts: 19
Quote:
Originally Posted by brianoz View Post
The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.
If you don't mind me asking:
- What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
- Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account? Did the user dump files in the system? Does a quick LMD scan reveal any PHP shells or other unwanted items?
- If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?

  #11  
Old
Web Hosting Master
 
Join Date: Nov 2004
Location: Australia
Posts: 1,529
Quote:
Originally Posted by unSpawn View Post
A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/"...
Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...

Quote:
Originally Posted by unSpawn View Post
If you don't mind me asking:
- What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
In other words, a standard shared server compromise where old WordPress/Joomla/etc installs or plugins are used to break into an account and run as that user.
Quote:
- Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account?
Haven't found an account, but a guess is no, as most user accounts had no shell.
Quote:
- If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?
Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.

  #12  
Old
Newbie
 
Join Date: Feb 2006
Posts: 19
Quote:
Originally Posted by brianoz View Post
Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...
Sorry, I misinterpreted what I saw. It should indeed be just /lib64/ or /lib/ like others said.


Quote:
Originally Posted by brianoz View Post
Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.
Sure, but still I'd rather have "evidence" or even a partial audit trail for analysis.

  #13  
Old
Web Hosting Master
 
Join Date: Jun 2001
Location: Princeton
Posts: 785
Anyone has details on the software / versions being installed on the server?
Something like rpm -qa from the servers would be very nice start.

__________________
Igor Seletskiy
CEO @ Cloud Linux Inc
http://www.cloudlinux.com
CloudLinux -- The OS that can make your Shared Hosting stable

  #14  
Old
Web Hosting Master
 
Join Date: Nov 2004
Location: Australia
Posts: 1,529
Output of rpm-qa: http://pastebin.com/nTc8wj3U

Output of rpm -Va (verify): http://pastebin.com/Fz0AxR3W
The "5" means modification, which is often benign, but may help. The -Va was run on a fully infected system, some changes may have been made by the time the -qa output was obtained.

See my post #2 above for matching O/S version etc:
Quote:
Originally Posted by brianoz View Post
CentOS release 6.3 (Final)
md5sum /lib64/libkeyutils.so.1.9
c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

-rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

rpm -qf /lib64/libkeyutils.so.1.9
file /lib64/libkeyutils.so.1.9 is not owned by any package

uname -r: 2.6.32-279.14.1.el6.x86_64.debug


Last edited by brianoz; 02-12-2013 at 09:40 PM. Reason: add OS version
  #15  
Old
Web Hosting Master
 
Join Date: Jun 2001
Location: Princeton
Posts: 785
Which control panel do affect servers run?
Anyone knows how they are getting infected yet?

__________________
Igor Seletskiy
CEO @ Cloud Linux Inc
http://www.cloudlinux.com
CloudLinux -- The OS that can make your Shared Hosting stable

Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
****`it Rootkit, Tuxtendo Rootkit ISpy Hosting Security and Technology 4 06-22-2010 11:27 AM
Which server builds are you rolling out? GeekMe Dedicated Server 11 04-18-2010 08:03 AM
Getting the ball rolling ... policefreq New Members 1 08-19-2006 11:16 PM
Getting company to get rolling Overclocked Running a Web Hosting Business 19 08-03-2004 04:02 PM

Related posts from TheWhir.com
Title Type Date Posted
Linux Malware Operation Windigo Infects 25,000 Web Servers Web Hosting News 2014-03-19 11:44:53
Hetzner Security Breach Exposes Customer Passwords, Payment Information Web Hosting News 2013-06-07 11:20:12
cPanel Addresses User Concerns of Transfer and Backup Restore System Security Web Hosting News 2013-05-24 10:13:44
Weekend Reading – Top Web Hosting News from the Week of February 19-22, 2013 Web Hosting News 2013-02-23 09:00:58
SSHD Rootkit in the Wild Blog 2013-02-22 16:44:08


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
WHT Host Brief Email:

We respect your privacy. We will never sell, rent, or give away your address to any outside party, ever.

Advertisement:
Web Hosting News:
WHT Membership
WHT Membership



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?