Page 1 of 61 12341151 ... LastLast
Results 1 to 25 of 1523
  1. #1
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,262

    SSHD Rootkit Rolling around

    Quick survey, anyone seen a rootkit being used to send spam through sshd involving a library called 'libkeyutils.so.1.9'?

    If so what OS did you see it on?
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
    Managed Servers (AS62710), Server Management, and Security Auditing.
    www.HostingSecList.com - Security notices for the hosting community.

  2. Thread Summary UPDATE (Feb 21): Adding fire to the local vulnerability theory, cPanel has just released the following statement. cPanel is not the cause of every rooted server obviously, but merely one of the avenues through which server credentials were stolen.

    You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.

    As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.
    UPDATE (Feb 21): Several Linux anti-malware scanners such as AVG now detect the malicious libkeyutils files based on signature instead of just name.

    UPDATE (Feb 20): Evidence is increasingly pointing towards a local vulnerability. The exploit filename also appears to be changing: libkeyutils-1.2.so.2 is popping up on CentOS 5.

    --

    If /lib64/libkeyutils.so.1.9 or /lib/libkeyutils.so.1.9 exist on your server, it is very likely that your server has been compromised at the root level and is currently sending out spam. Removing this file may be a temporary fix, but since the attack vector is still unknown, that is not likely a permanent fix. At this point, if your server has been rooted, the only 100% way to clean your server is to wipe your drives and do a clean installation.

    Possibilities being discussed in this thread include a 0-day exploit of SSHD itself, curl vulnerabilities or even a local vulnerability attacking users through software like Adobe Flash and gaining root access to their servers via their computers.

    Based on community input, it appears that both RHEL-based and Debian servers are affected. Servers with control panels such as cPanel, DirectAdmin, and Plesk are also affected. Servers with both standard and non-standard SSH ports are vulnerable and even servers that only accept key authentication have been compromised. Consider all passwords (including root) and private/public keys compromised. If you've made SSH connections to other servers from your exploited server, that login information is likely also compromised.

    Recommended Actions: Since we still do not know the attack vector, we can only provide guidelines for things you should probably do.

    • Change all of your root passwords and key pairs from a clean computer
    • Keep your server software up-to-date
    • Disable root logins and/or firewall off your SSH port
    • Upgrade Flash and Java on your computers
    • Do malware scans on your computers
    • Keep checking this thread for updates! This thread summary will be constantly updated when we have new information.


    WARNING: There are multiple scripts floating around the internet that promise to automatically clean up your server, but please be aware that they are not guaranteed to fix anything and have the potential to cause more problems. Run them at your own risk!

    Contributors: Orien

  3. #2
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,604
    The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug

  4. #3
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    I too can confirm this. Currently working with clients with spam issues and it is present. I checked other boxes we run and own and the library is no where to be found. It is only found on spam infested machines.

    uname -a
    2.6.32-042stab059.7

    md5sum /lib64/libkeyutils.so.1.9
    d81217186da61125f4dad7a87857b697 /lib64/libkeyutils.so.1.9

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

  5. #4
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,262
    Quote Originally Posted by brianoz View Post
    The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug

    They are not logging in with root, nor are they even spawning a bash process.
    If the lib is moved out, and sshd is restarted they cannot login anymore fwiw.

    The key is finding out how they are getting in. Fully upgraded, ssh key restricted sshd, on non-standard ports are being compromised.
    None of my customers are, but I have been getting alot of sales inquiries with this issue so I don't know the full history of the machines.

    Seeing it on centos 5, centos 6, cloudlinux 5, cloudlinux 6.
    Last edited by Steven; 02-08-2013 at 01:27 PM.
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
    Managed Servers (AS62710), Server Management, and Security Auditing.
    www.HostingSecList.com - Security notices for the hosting community.

  6. #5
    Join Date
    Sep 2000
    Location
    New Jersey
    Posts
    389
    I haven't seen this yet, but will keep my eye out.

    64 bit only systems are what you are seeing? Are tcpwrappers or firewall offing ssh making a difference?
    John Quaglieri - InterServer, Inc

  7. #6
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,604
    Firewalling off ssh stopped them on the machine I was looking at.
    And the machine was 64 bit.

    FWIW I suspect they are getting in initially some other way than ssh, but have no evidence.

  8. #7
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,262
    John,
    Iptables will stop them.
    Tcpwrappers does not.

    Brianoz,
    Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
    Managed Servers (AS62710), Server Management, and Security Auditing.
    www.HostingSecList.com - Security notices for the hosting community.

  9. #8
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,604
    Quote Originally Posted by Steven View Post
    Brianoz,
    Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.
    Steven - sorry for being unclear. I didn't mean to imply the initial attack was from ssh; what I meant was that an iptables block on ssh stopped them reconnecting, exactly as you've seen.

    Is the following consistent with what you've seen?

    1. User account compromised at PHP level
    2. Compromised account used to hack root and backdoor sshd via libkeyutils
    3. Spam sent

    The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.

  10. #9
    A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/". Would anyone of you be able to dump a copy of the file(s) at sourceforge +.net/tracker/ +?group_id=155034&atid=794187 or send it to me for analysis? Much appreciated, TIA.
    Last edited by unSpawn; 02-09-2013 at 09:28 AM. Reason: //More *is* more

  11. #10
    Quote Originally Posted by brianoz View Post
    The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.
    If you don't mind me asking:
    - What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
    - Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account? Did the user dump files in the system? Does a quick LMD scan reveal any PHP shells or other unwanted items?
    - If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?

  12. #11
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,604
    Quote Originally Posted by unSpawn View Post
    A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/"...
    Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...

    Quote Originally Posted by unSpawn View Post
    If you don't mind me asking:
    - What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
    In other words, a standard shared server compromise where old WordPress/Joomla/etc installs or plugins are used to break into an account and run as that user.
    - Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account?
    Haven't found an account, but a guess is no, as most user accounts had no shell.
    - If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?
    Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.

  13. #12
    Quote Originally Posted by brianoz View Post
    Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...
    Sorry, I misinterpreted what I saw. It should indeed be just /lib64/ or /lib/ like others said.


    Quote Originally Posted by brianoz View Post
    Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.
    Sure, but still I'd rather have "evidence" or even a partial audit trail for analysis.

  14. #13
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    815
    Anyone has details on the software / versions being installed on the server?
    Something like rpm -qa from the servers would be very nice start.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  15. #14
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,604
    Output of rpm-qa: http://pastebin.com/nTc8wj3U

    Output of rpm -Va (verify): http://pastebin.com/Fz0AxR3W
    The "5" means modification, which is often benign, but may help. The -Va was run on a fully infected system, some changes may have been made by the time the -qa output was obtained.

    See my post #2 above for matching O/S version etc:
    Quote Originally Posted by brianoz View Post
    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug
    Last edited by brianoz; 02-12-2013 at 09:40 PM. Reason: add OS version

  16. #15
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    815
    Which control panel do affect servers run?
    Anyone knows how they are getting infected yet?
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  17. #16
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    Quote Originally Posted by iseletsk View Post
    Which control panel do affect servers run?
    Anyone knows how they are getting infected yet?
    All the ones we have seen so far are CPanel and they have been poorly secured to begin with.

  18. #17
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,604
    cPanel, and also poorly secured. We don't know how they are getting to root to install the backdoor yet.

  19. #18
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,262
    Trying to tackle all angles, what imap/pop3 server are you seeing on the servers (dovecot vs courier)?
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
    Managed Servers (AS62710), Server Management, and Security Auditing.
    www.HostingSecList.com - Security notices for the hosting community.

  20. #19
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,604
    FYI, courier on the only server I know that was exploited.

  21. #20
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,262
    What about you SolidShellSecurity ?
    Could be off base here, but I have not seen a server with dovecot exploited.
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    System Administration Extraordinaire | Follow us on twitter:@Rack911Labs
    Managed Servers (AS62710), Server Management, and Security Auditing.
    www.HostingSecList.com - Security notices for the hosting community.

  22. #21
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,873
    Been so long since a Courier IMAP exploit existed... What FTP daemon were all of those boxes running?

  23. #22
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    Quote Originally Posted by Steven View Post
    What about you SolidShellSecurity ?
    Could be off base here, but I have not seen a server with dovecot exploited.
    I don't exactly remember. I would check but the box would be wiped by now as we moved them off to a new server. But I vaguely don't remember them running dovecot.

  24. #23
    Join Date
    Sep 2000
    Location
    New Jersey
    Posts
    389
    Just with the fact only 64 bit servers (so far) are known to be exploited, it could be related to past exploits on 64 bit systems. Its been a while there, but I wouldn't discount previously hacked machines from that kind of exploit.
    John Quaglieri - InterServer, Inc

  25. #24
    Join Date
    May 2003
    Location
    Texas
    Posts
    148
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    Tired of being told your request is "unsupported"? Look no more!
    Fully Managed Dedicated servers, Virtual Private Servers, Shared Hosting & more!
    HostDigit.com - "The" Managed Provider! WE DEFINE MANAGED HOSTING
    Up to 256 IP addresses per server! Let us be your admin!

  26. #25
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,873
    Quote Originally Posted by Majester View Post
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    When a server is deemed compromised, it's always a good idea to do a check of all non-user directories to look for recently changed binaries or anything new. My guess, that file showed up in the search.
    Patrick William | RACK911 Labs | Software Security Auditing
    400+ Vulnerabilities Found - Free Quote @ https://www.RACK911Labs.com

    www.HostingSecList.com - Security notices for the hosting community.

Page 1 of 61 12341151 ... LastLast

Similar Threads

  1. ****`it Rootkit, Tuxtendo Rootkit
    By ISpy in forum Hosting Security and Technology
    Replies: 4
    Last Post: 06-22-2010, 11:27 AM
  2. Which server builds are you rolling out?
    By GeekMe in forum Dedicated Server
    Replies: 11
    Last Post: 04-18-2010, 08:03 AM
  3. Getting the ball rolling ...
    By policefreq in forum New Members
    Replies: 1
    Last Post: 08-19-2006, 11:16 PM
  4. Getting company to get rolling
    By Overclocked in forum Running a Web Hosting Business
    Replies: 19
    Last Post: 08-03-2004, 04:02 PM

Related Posts from theWHIR.com

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •