Page 1 of 61 12341151 ... LastLast
Results 1 to 25 of 1523
  1. #1
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681

    SSHD Rootkit Rolling around

    Quick survey, anyone seen a rootkit being used to send spam through sshd involving a library called 'libkeyutils.so.1.9'?

    If so what OS did you see it on?
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  2. #2
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug

  3. #3
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,954
    I too can confirm this. Currently working with clients with spam issues and it is present. I checked other boxes we run and own and the library is no where to be found. It is only found on spam infested machines.

    uname -a
    2.6.32-042stab059.7

    md5sum /lib64/libkeyutils.so.1.9
    d81217186da61125f4dad7a87857b697 /lib64/libkeyutils.so.1.9

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

  4. #4
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by brianoz View Post
    The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug

    They are not logging in with root, nor are they even spawning a bash process.
    If the lib is moved out, and sshd is restarted they cannot login anymore fwiw.

    The key is finding out how they are getting in. Fully upgraded, ssh key restricted sshd, on non-standard ports are being compromised.
    None of my customers are, but I have been getting alot of sales inquiries with this issue so I don't know the full history of the machines.

    Seeing it on centos 5, centos 6, cloudlinux 5, cloudlinux 6.
    Last edited by Steven; 02-08-2013 at 01:27 PM.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  5. #5
    Join Date
    Sep 2000
    Posts
    428
    I haven't seen this yet, but will keep my eye out.

    64 bit only systems are what you are seeing? Are tcpwrappers or firewall offing ssh making a difference?

  6. #6
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    Firewalling off ssh stopped them on the machine I was looking at.
    And the machine was 64 bit.

    FWIW I suspect they are getting in initially some other way than ssh, but have no evidence.

  7. #7
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    John,
    Iptables will stop them.
    Tcpwrappers does not.

    Brianoz,
    Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  8. #8
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    Quote Originally Posted by Steven View Post
    Brianoz,
    Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.
    Steven - sorry for being unclear. I didn't mean to imply the initial attack was from ssh; what I meant was that an iptables block on ssh stopped them reconnecting, exactly as you've seen.

    Is the following consistent with what you've seen?

    1. User account compromised at PHP level
    2. Compromised account used to hack root and backdoor sshd via libkeyutils
    3. Spam sent

    The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.

  9. #9
    A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/". Would anyone of you be able to dump a copy of the file(s) at sourceforge +.net/tracker/ +?group_id=155034&atid=794187 or send it to me for analysis? Much appreciated, TIA.
    Last edited by unSpawn; 02-09-2013 at 09:28 AM. Reason: //More *is* more

  10. #10
    Quote Originally Posted by brianoz View Post
    The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.
    If you don't mind me asking:
    - What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
    - Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account? Did the user dump files in the system? Does a quick LMD scan reveal any PHP shells or other unwanted items?
    - If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?

  11. #11
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    Quote Originally Posted by unSpawn View Post
    A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/"...
    Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...

    Quote Originally Posted by unSpawn View Post
    If you don't mind me asking:
    - What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
    In other words, a standard shared server compromise where old WordPress/Joomla/etc installs or plugins are used to break into an account and run as that user.
    - Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account?
    Haven't found an account, but a guess is no, as most user accounts had no shell.
    - If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?
    Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.

  12. #12
    Quote Originally Posted by brianoz View Post
    Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...
    Sorry, I misinterpreted what I saw. It should indeed be just /lib64/ or /lib/ like others said.


    Quote Originally Posted by brianoz View Post
    Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.
    Sure, but still I'd rather have "evidence" or even a partial audit trail for analysis.

  13. #13
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    1,029
    Anyone has details on the software / versions being installed on the server?
    Something like rpm -qa from the servers would be very nice start.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  14. #14
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    Output of rpm-qa: http://pastebin.com/nTc8wj3U

    Output of rpm -Va (verify): http://pastebin.com/Fz0AxR3W
    The "5" means modification, which is often benign, but may help. The -Va was run on a fully infected system, some changes may have been made by the time the -qa output was obtained.

    See my post #2 above for matching O/S version etc:
    Quote Originally Posted by brianoz View Post
    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug
    Last edited by brianoz; 02-12-2013 at 09:40 PM. Reason: add OS version

  15. #15
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    1,029
    Which control panel do affect servers run?
    Anyone knows how they are getting infected yet?
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  16. #16
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,954
    Quote Originally Posted by iseletsk View Post
    Which control panel do affect servers run?
    Anyone knows how they are getting infected yet?
    All the ones we have seen so far are CPanel and they have been poorly secured to begin with.

  17. #17
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    cPanel, and also poorly secured. We don't know how they are getting to root to install the backdoor yet.

  18. #18
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Trying to tackle all angles, what imap/pop3 server are you seeing on the servers (dovecot vs courier)?
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  19. #19
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    FYI, courier on the only server I know that was exploited.

  20. #20
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    What about you SolidShellSecurity ?
    Could be off base here, but I have not seen a server with dovecot exploited.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  21. #21
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Been so long since a Courier IMAP exploit existed... What FTP daemon were all of those boxes running?

  22. #22
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,954
    Quote Originally Posted by Steven View Post
    What about you SolidShellSecurity ?
    Could be off base here, but I have not seen a server with dovecot exploited.
    I don't exactly remember. I would check but the box would be wiped by now as we moved them off to a new server. But I vaguely don't remember them running dovecot.

  23. #23
    Join Date
    Sep 2000
    Posts
    428
    Just with the fact only 64 bit servers (so far) are known to be exploited, it could be related to past exploits on 64 bit systems. Its been a while there, but I wouldn't discount previously hacked machines from that kind of exploit.

  24. #24
    Join Date
    May 2003
    Location
    Texas
    Posts
    154
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    DDoS Protected Chicago and New York Virtual Private Servers with INSTANT setup!
    RAID-10 OpenVZ Virtual Private Servers with hundreds of OS templates!
    CometVPS.com - We're all about customer experience. Try us!

  25. #25
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Quote Originally Posted by Majester View Post
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    When a server is deemed compromised, it's always a good idea to do a check of all non-user directories to look for recently changed binaries or anything new. My guess, that file showed up in the search.
    RACK911 Labs | Penetration Testing | https://www.RACK911Labs.ca

    www.HostingSecList.com - Security Notices for the Hosting Community.

Page 1 of 61 12341151 ... LastLast

Similar Threads

  1. ****`it Rootkit, Tuxtendo Rootkit
    By ISpy in forum Hosting Security and Technology
    Replies: 4
    Last Post: 06-22-2010, 11:27 AM
  2. Which server builds are you rolling out?
    By GeekMe in forum Dedicated Server
    Replies: 11
    Last Post: 04-18-2010, 08:03 AM
  3. Getting the ball rolling ...
    By policefreq in forum New Members
    Replies: 1
    Last Post: 08-19-2006, 11:16 PM
  4. Getting company to get rolling
    By Overclocked in forum Running a Web Hosting Business
    Replies: 19
    Last Post: 08-03-2004, 04:02 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •