Page 1 of 39 1234 11 ... LastLast
Results 1 to 40 of 1523
  1. #1
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267

    SSHD Rootkit Rolling around

    Quick survey, anyone seen a rootkit being used to send spam through sshd involving a library called 'libkeyutils.so.1.9'?

    If so what OS did you see it on?
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    Managed Servers (AS62710), Server Management, and Security Auditing.

  2. Thread Summary UPDATE (Feb 21): Adding fire to the local vulnerability theory, cPanel has just released the following statement. cPanel is not the cause of every rooted server obviously, but merely one of the avenues through which server credentials were stolen.

    You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.

    As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.
    UPDATE (Feb 21): Several Linux anti-malware scanners such as AVG now detect the malicious libkeyutils files based on signature instead of just name.

    UPDATE (Feb 20): Evidence is increasingly pointing towards a local vulnerability. The exploit filename also appears to be changing: libkeyutils-1.2.so.2 is popping up on CentOS 5.

    --

    If /lib64/libkeyutils.so.1.9 or /lib/libkeyutils.so.1.9 exist on your server, it is very likely that your server has been compromised at the root level and is currently sending out spam. Removing this file may be a temporary fix, but since the attack vector is still unknown, that is not likely a permanent fix. At this point, if your server has been rooted, the only 100% way to clean your server is to wipe your drives and do a clean installation.

    Possibilities being discussed in this thread include a 0-day exploit of SSHD itself, curl vulnerabilities or even a local vulnerability attacking users through software like Adobe Flash and gaining root access to their servers via their computers.

    Based on community input, it appears that both RHEL-based and Debian servers are affected. Servers with control panels such as cPanel, DirectAdmin, and Plesk are also affected. Servers with both standard and non-standard SSH ports are vulnerable and even servers that only accept key authentication have been compromised. Consider all passwords (including root) and private/public keys compromised. If you've made SSH connections to other servers from your exploited server, that login information is likely also compromised.

    Recommended Actions: Since we still do not know the attack vector, we can only provide guidelines for things you should probably do.

    • Change all of your root passwords and key pairs from a clean computer
    • Keep your server software up-to-date
    • Disable root logins and/or firewall off your SSH port
    • Upgrade Flash and Java on your computers
    • Do malware scans on your computers
    • Keep checking this thread for updates! This thread summary will be constantly updated when we have new information.


    WARNING: There are multiple scripts floating around the internet that promise to automatically clean up your server, but please be aware that they are not guaranteed to fix anything and have the potential to cause more problems. Run them at your own risk!

    Contributors: Orien

  3. #2
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,677
    The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug

  4. #3
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    I too can confirm this. Currently working with clients with spam issues and it is present. I checked other boxes we run and own and the library is no where to be found. It is only found on spam infested machines.

    uname -a
    2.6.32-042stab059.7

    md5sum /lib64/libkeyutils.so.1.9
    d81217186da61125f4dad7a87857b697 /lib64/libkeyutils.so.1.9

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

  5. #4
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267
    Quote Originally Posted by brianoz View Post
    The "sending spam through sshd" part sounds familiar, and /lib64/libkeyutils.so.1.9 is present on the hacked system but not on other Centos 6.3 servers. The techs (unreliable?) reported a root login wasn't prevented by a password change.

    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug

    They are not logging in with root, nor are they even spawning a bash process.
    If the lib is moved out, and sshd is restarted they cannot login anymore fwiw.

    The key is finding out how they are getting in. Fully upgraded, ssh key restricted sshd, on non-standard ports are being compromised.
    None of my customers are, but I have been getting alot of sales inquiries with this issue so I don't know the full history of the machines.

    Seeing it on centos 5, centos 6, cloudlinux 5, cloudlinux 6.
    Last edited by Steven; 02-08-2013 at 01:27 PM.
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    Managed Servers (AS62710), Server Management, and Security Auditing.

  6. #5
    Join Date
    Sep 2000
    Location
    New Jersey
    Posts
    389
    I haven't seen this yet, but will keep my eye out.

    64 bit only systems are what you are seeing? Are tcpwrappers or firewall offing ssh making a difference?
    John Quaglieri - InterServer, Inc

  7. #6
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,677
    Firewalling off ssh stopped them on the machine I was looking at.
    And the machine was 64 bit.

    FWIW I suspect they are getting in initially some other way than ssh, but have no evidence.

  8. #7
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267
    John,
    Iptables will stop them.
    Tcpwrappers does not.

    Brianoz,
    Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    Managed Servers (AS62710), Server Management, and Security Auditing.

  9. #8
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,677
    Quote Originally Posted by Steven View Post
    Brianoz,
    Its unlikely its ssh, found a box that had the file but ssh was disabled with a hw firewall.
    Steven - sorry for being unclear. I didn't mean to imply the initial attack was from ssh; what I meant was that an iptables block on ssh stopped them reconnecting, exactly as you've seen.

    Is the following consistent with what you've seen?

    1. User account compromised at PHP level
    2. Compromised account used to hack root and backdoor sshd via libkeyutils
    3. Spam sent

    The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.

  10. #9
    A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/". Would anyone of you be able to dump a copy of the file(s) at sourceforge +.net/tracker/ +?group_id=155034&atid=794187 or send it to me for analysis? Much appreciated, TIA.
    Last edited by unSpawn; 02-09-2013 at 09:28 AM. Reason: //More *is* more

  11. #10
    Quote Originally Posted by brianoz View Post
    The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.
    If you don't mind me asking:
    - What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
    - Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account? Did the user dump files in the system? Does a quick LMD scan reveal any PHP shells or other unwanted items?
    - If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?

  12. #11
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,677
    Quote Originally Posted by unSpawn View Post
    A quick search showed a couple of web hosts, cleaned up now apparently, where any leading (doc root?) directory names include "sym" and "lib" (as in /%{accountname}/sym/lib%{arch}) but most often "/sym/root/usr/lib%{arch}/"...
    Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...

    Quote Originally Posted by unSpawn View Post
    If you don't mind me asking:
    - What do you exactly mean with "account compromised at PHP level"? Do you mean the attacker leveraged a known vulnerability in a product or is it a guess?
    In other words, a standard shared server compromise where old WordPress/Joomla/etc installs or plugins are used to break into an account and run as that user.
    - Did this compromised account have a valid shell? Does its shell history show any "interesting" commands like wget, cURL or other downloads? Do system or daemon logs show any commands related to this users account?
    Haven't found an account, but a guess is no, as most user accounts had no shell.
    - If you trawl your logs, could you guesstimate how much time there would have been approximately between the initial breach and the root compromise?
    Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.

  13. #12
    Quote Originally Posted by brianoz View Post
    Could you state the names of the files you're looking for a little more clearly? Couldn't see anything like /home/*/sym/lib or /home/*/sym/usr/lib ...
    Sorry, I misinterpreted what I saw. It should indeed be just /lib64/ or /lib/ like others said.


    Quote Originally Posted by brianoz View Post
    Given the number of servers reporting this, I'm making an educated guess that it has to be at least partly automated.
    Sure, but still I'd rather have "evidence" or even a partial audit trail for analysis.

  14. #13
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    827
    Anyone has details on the software / versions being installed on the server?
    Something like rpm -qa from the servers would be very nice start.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  15. #14
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,677
    Output of rpm-qa: http://pastebin.com/nTc8wj3U

    Output of rpm -Va (verify): http://pastebin.com/Fz0AxR3W
    The "5" means modification, which is often benign, but may help. The -Va was run on a fully infected system, some changes may have been made by the time the -qa output was obtained.

    See my post #2 above for matching O/S version etc:
    Quote Originally Posted by brianoz View Post
    CentOS release 6.3 (Final)
    md5sum /lib64/libkeyutils.so.1.9
    c1f53b3ecb05102d46f1d533fe093529 /lib64/libkeyutils.so.1.9

    -rwxr-xr-x 1 root root 34584 Jun 22 2012 /lib64/libkeyutils.so.1.9*

    rpm -qf /lib64/libkeyutils.so.1.9
    file /lib64/libkeyutils.so.1.9 is not owned by any package

    uname -r: 2.6.32-279.14.1.el6.x86_64.debug
    Last edited by brianoz; 02-12-2013 at 09:40 PM. Reason: add OS version

  16. #15
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    827
    Which control panel do affect servers run?
    Anyone knows how they are getting infected yet?
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  17. #16
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    Quote Originally Posted by iseletsk View Post
    Which control panel do affect servers run?
    Anyone knows how they are getting infected yet?
    All the ones we have seen so far are CPanel and they have been poorly secured to begin with.

  18. #17
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,677
    cPanel, and also poorly secured. We don't know how they are getting to root to install the backdoor yet.

  19. #18
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267
    Trying to tackle all angles, what imap/pop3 server are you seeing on the servers (dovecot vs courier)?
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    Managed Servers (AS62710), Server Management, and Security Auditing.

  20. #19
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,677
    FYI, courier on the only server I know that was exploited.

  21. #20
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267
    What about you SolidShellSecurity ?
    Could be off base here, but I have not seen a server with dovecot exploited.
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    Managed Servers (AS62710), Server Management, and Security Auditing.

  22. #21
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,908
    Been so long since a Courier IMAP exploit existed... What FTP daemon were all of those boxes running?

  23. #22
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    Quote Originally Posted by Steven View Post
    What about you SolidShellSecurity ?
    Could be off base here, but I have not seen a server with dovecot exploited.
    I don't exactly remember. I would check but the box would be wiped by now as we moved them off to a new server. But I vaguely don't remember them running dovecot.

  24. #23
    Join Date
    Sep 2000
    Location
    New Jersey
    Posts
    389
    Just with the fact only 64 bit servers (so far) are known to be exploited, it could be related to past exploits on 64 bit systems. Its been a while there, but I wouldn't discount previously hacked machines from that kind of exploit.
    John Quaglieri - InterServer, Inc

  25. #24
    Join Date
    May 2003
    Location
    Texas
    Posts
    149
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    DDoS Protected Chicago and New York Virtual Private Servers with INSTANT setup!
    RAID-10 OpenVZ Virtual Private Servers with hundreds of OS templates!
    CometVPS.com - "The" definition of personable support, that's us!
    Use coupon "50FORLIFE" for 50% off the life of ANY plan!

  26. #25
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,908
    Quote Originally Posted by Majester View Post
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    When a server is deemed compromised, it's always a good idea to do a check of all non-user directories to look for recently changed binaries or anything new. My guess, that file showed up in the search.
    Patrick William | RACK911 Labs | Software Security Auditing
    400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com

    www.HostingSecList.com - Security notices for the hosting community.

  27. #26
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267
    Quote Originally Posted by Majester View Post
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    Alot of debugging. Started with lsof of running ssh binaries, and verified the libraries involved were actually attached to a RPM package.
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    Managed Servers (AS62710), Server Management, and Security Auditing.

  28. #27
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267
    Quote Originally Posted by Patrick View Post
    When a server is deemed compromised, it's always a good idea to do a check of all non-user directories to look for recently changed binaries or anything new. My guess, that file showed up in the search.
    Quote Originally Posted by Majester View Post
    How did you come to the conclusion on finding /lib64/libkeyutils.so.1.9 in the first place? What led you to this file?
    This is how I found it:

    [email protected] [~]# rpm -qf `lsof -p 785953 | grep lib | awk '{print $9}'`
    glibc-2.12-1.80.el6_3.7.x86_64
    nspr-4.9.2-0.el6_3.1.x86_64
    nspr-4.9.2-0.el6_3.1.x86_64
    nspr-4.9.2-0.el6_3.1.x86_64
    nss-util-3.13.6-1.el6_3.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    file /lib64/libkeyutils.so.1.9 is not owned by any package
    krb5-libs-1.9-33.el6_3.3.x86_64
    nss-softokn-freebl-3.12.9-11.el6.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    nss-3.13.5-1.el6_3.x86_64
    libcom_err-1.41.12-12.el6.x86_64
    krb5-libs-1.9-33.el6_3.3.x86_64
    krb5-libs-1.9-33.el6_3.3.x86_64
    krb5-libs-1.9-33.el6_3.3.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    zlib-1.2.3-27.el6.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    openssl-1.0.0-25.el6_3.1.x86_64
    libselinux-2.0.94-5.3.el6.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    pam-1.1.1-10.el6_2.1.x86_64
    audit-libs-2.2-2.el6.x86_64
    tcp_wrappers-libs-7.6-57.el6.x86_64
    fipscheck-lib-1.2.0-7.el6.x86_64
    glibc-2.12-1.80.el6_3.7.x86_64
    [email protected] [~]#
    Steven Ciaburri | Proactive Linux Server Management - Rack911.com
    Managed Servers (AS62710), Server Management, and Security Auditing.

  29. #28
    Join Date
    Jan 2013
    Posts
    379
    Many thanks Steven. Do we have hunter for this root kit ?

  30. #29
    Maybe LMD but not CRT and definitely not RKH until somebody drops the payload like I asked in post #9,
    TIA.

  31. #30
    Join Date
    Jan 2013
    Posts
    379
    It is complete crap.

    If you remove libkeyutils.so.1.9, it won't start neither nsd or ssh

  32. #31
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    827
    mobilk: remove that file
    do: yum reinstall keyutils-libs

    of course the question on how hackers got in is still open.
    and btw: if you use passwords to login, change them.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  33. #32
    Join Date
    Jan 2013
    Posts
    379
    Quote Originally Posted by iseletsk View Post
    mobilk: remove that file
    do: yum reinstall keyutils-libs
    Thanks!. BTW compromised system was CloudLinux one. Just installed 1 month ago.

  34. #33
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    827
    Yes, there are few CL ones, Steven told me. So far, from what we know it affects CL & CentOS systems running cPanel. No idea yet on how hackers get in.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  35. #34
    Join Date
    Nov 2002
    Location
    Oklahoma
    Posts
    591
    Quote Originally Posted by iseletsk View Post
    Yes, there are few CL ones, Steven told me. So far, from what we know it affects CL & CentOS systems running cPanel. No idea yet on how hackers get in.
    Were any of the compromised CL machines using CageFS as well? Obviously it may not matter depending on the point of entry but curious nonetheless.
    Dathorn, Inc. - Premium WHM & cPanel Hosting since 2002! Check Out Our Blog!
    Experience the Dathorn Difference! - [email protected]
    LiteSpeed | Clustered DNS | CloudLinux | CageFS | KernelCare
    Pure SSD Storage | R1Soft Daily Backups | Softaculous | SpamTitan

  36. #35
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    We haven't seen any of our GRSecurity servers hit yet. Anyone else seen a GRSecurity server hit? It might help to pinpoint a kernel vuln.

  37. #36
    Join Date
    Jan 2013
    Posts
    379
    Quote Originally Posted by Dathorn-Andrew View Post
    Were any of the compromised CL machines using CageFS as well? Obviously it may not matter depending on the point of entry but curious nonetheless.
    Yes, it was.

  38. #37
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,565
    Have you guys checked the WHM logs, perhaps correlating with the timestamp on the file? Were the kernels reasonably up to date on the affected systems?
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  39. #38
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,748
    Quote Originally Posted by FastServ View Post
    Have you guys checked the WHM logs, perhaps correlating with the timestamp on the file? Were the kernels reasonably up to date on the affected systems?
    On that subject, what WHM plugins are running? If every server has in someway been cpanel related maybe they found a hole in using one of those? CPanel does not exactly do a good job with securing this area as they all run as root. The servers we have seen have been running a lot of plugins.

  40. #39
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,565
    Quote Originally Posted by TravisT-[SSS] View Post
    On that subject, what WHM plugins are running? If every server has in someway been cpanel related maybe they found a hole in using one of those? CPanel does not exactly do a good job with securing this area as they all run as root. The servers we have seen have been running a lot of plugins.
    Which is along the line of my thinking. Two most common way in my experience to drop root files would be either thru WHM (don't we all love how WHM and all its 3rd party plugins runs as root), or privilege escalation on an ancient kernel. It's also a good idea to make sure WHM has been updating itself properly.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  41. #40
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,267
    From a number of servers I investigated since I first saw this.. no whm login nor old kernel on most of them.

Page 1 of 39 1234 11 ... LastLast

Similar Threads

  1. ****`it Rootkit, Tuxtendo Rootkit
    By ISpy in forum Hosting Security and Technology
    Replies: 4
    Last Post: 06-22-2010, 11:27 AM
  2. Which server builds are you rolling out?
    By GeekMe in forum Dedicated Server
    Replies: 11
    Last Post: 04-18-2010, 08:03 AM
  3. Getting the ball rolling ...
    By policefreq in forum New Members
    Replies: 1
    Last Post: 08-19-2006, 11:16 PM
  4. Getting company to get rolling
    By Overclocked in forum Running a Web Hosting Business
    Replies: 19
    Last Post: 08-03-2004, 04:02 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •