Page 11 of 61 FirstFirst ... 89101112131421 ... LastLast
Results 251 to 275 of 1523
  1. #251
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Quote Originally Posted by jalapeno55 View Post
    I believe that file is normal, I have it as well:
    Code:
    # for i in `ps aux  |grep ssh |grep -v grep |awk {'print $2'}`;do rpm -qf `lsof -p $i | grep lib | awk '{print $9}'`;done |sort -n |uniq
    file /lib64/security/pam_hulk.so is not owned by any package
    Yeah it's legit. I think people need to stop grasping at straws and coming to conclusions without knowing what they are looking at to make such a conclusion.

  2. #252
    Join Date
    Feb 2013
    Posts
    97
    If so, what is your output of strings /lib64/security/pam_hulk.so please?

  3. #253
    Join Date
    Mar 2012
    Posts
    57
    Pam_hulk is fine that is cpanel's brute force module, which is not installed via yum so will give a warning.

  4. #254
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Quote Originally Posted by M Bacon View Post
    22-Jun-2012 02:20:37
    What's interesting is that the RPM was built on that day:

    http://rpmfind.net/linux/RPM/centos/....el6.i686.html

  5. #255
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Quote Originally Posted by matbz View Post
    If so, what is your output of strings /lib64/security/pam_hulk.so please?
    Same as yours:

    IO7+
    __gmon_start__
    _init
    _fini
    __cxa_finalize
    _Jv_RegisterClasses
    pam_hulk_version
    openlog
    vsyslog
    closelog
    pam_get_item
    readline
    fileno
    read
    feof
    get_hulk_key
    fopen
    fclose
    disconnect_hulk
    memset
    sigfillset
    sigaction
    alarm
    socket
    memcpy
    connect
    snprintf
    strlen
    write
    tell_hulk
    strncmp
    hulk_enabled
    pam_sm_authenticate
    pam_get_user
    time
    pam_sm_setcred
    libc.so.6
    __xstat
    libpam.so.0
    _edata
    __bss_start
    _end
    LIBPAM_1.0
    GLIBC_2.2.5
    ATSubH
    cphulk_version_2.7
    PAM-hulk
    timeout while connecting to cphulkd
    /var/cpanel/cphulkd/keys/pam
    error getting hulk key
    error setting sigalrm
    opening socket
    /var/run/cphulkd.sock
    failed to connect stream socket
    error reading welcome from stream socket
    Unexpected welcome messge from cphulkd: %s
    AUTH pam %s
    error reading auth status from stream socket
    error logging into hulkd: %s
    PAM_AUTHENTICATE system %s %s 1 %ld 0 0 %s
    PAM_SETCRED system %s %s 1 %ld 0 0 %s
    err writing to stream socket
    error reading PAM_? from stream socket
    Brute force detection active: %s
    /var/cpanel/hulkd/enabled
    /var/cpanel/cphulk_enable
    crond

  6. #256
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    1,029
    I think pam_hulk is part of cPanel hulkd, and not a hack at all.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  7. #257
    I believe that file exists on previous hacked servers. Please confirm.

  8. #258
    Join Date
    Feb 2013
    Posts
    97
    Quote Originally Posted by Patrick View Post
    Same as yours:

    IO7+
    __gmon_start__
    _init
    _fini
    __cxa_finalize
    _Jv_RegisterClasses
    pam_hulk_version
    openlog
    vsyslog
    closelog
    pam_get_item
    readline
    fileno
    read
    feof
    get_hulk_key
    fopen
    fclose
    disconnect_hulk
    memset
    sigfillset
    sigaction
    alarm
    socket
    memcpy
    connect
    snprintf
    strlen
    write
    tell_hulk
    strncmp
    hulk_enabled
    pam_sm_authenticate
    pam_get_user
    time
    pam_sm_setcred
    libc.so.6
    __xstat
    libpam.so.0
    _edata
    __bss_start
    _end
    LIBPAM_1.0
    GLIBC_2.2.5
    ATSubH
    cphulk_version_2.7
    PAM-hulk
    timeout while connecting to cphulkd
    /var/cpanel/cphulkd/keys/pam
    error getting hulk key
    error setting sigalrm
    opening socket
    /var/run/cphulkd.sock
    failed to connect stream socket
    error reading welcome from stream socket
    Unexpected welcome messge from cphulkd: %s
    AUTH pam %s
    error reading auth status from stream socket
    error logging into hulkd: %s
    PAM_AUTHENTICATE system %s %s 1 %ld 0 0 %s
    PAM_SETCRED system %s %s 1 %ld 0 0 %s
    err writing to stream socket
    error reading PAM_? from stream socket
    Brute force detection active: %s
    /var/cpanel/hulkd/enabled
    /var/cpanel/cphulk_enable
    crond
    thnx, confirmed on 6 other boxes.

  9. #259
    Join Date
    Mar 2012
    Location
    Tampa, FL =)
    Posts
    1,954
    We can confirm that the pam module used by CPanel is not been exploited. We have the same strings match up on a clean box not being used to do spam nor have we seen any activity hinting to an exploit.

    I think it would be safe if we don't jump to conclusions here without posting verbose logs from snoopy ect.. to backup findings.

    On a sidenote, I think it would be a good idea if everyone was running snoopy till we find the root cause.

    On a random note: I think it is safe to say though we should expect the attackers to start randomizing the file at some point to attempt to remain more hidden or change their tactics up some.
    Last edited by TravisT-[SSS]; 02-18-2013 at 03:58 PM.

  10. #260
    Join Date
    Apr 2008
    Location
    Tulsa, OK, USA
    Posts
    376
    Quote Originally Posted by matbz View Post
    This made me feel very uncomfortable...

    Code:
    root@titanium [~]# ps aux | grep ssh
    root      1667  0.0  0.0  61192   752 pts/0    S+   19:29   0:00 grep ssh
    root      7722  0.0  0.0  92244  3416 ?        Ss   15:58   0:00 sshd: root@pts/0
    root     21883  0.0  0.0  63584  1212 ?        Ss   Jan17   0:00 /usr/sbin/sshd

    Code:
    root@titanium [~]# rpm -qf `lsof -p 7722 | grep lib | awk '{print $9}'`
    glibc-2.5-107
    tcp_wrappers-7.6-40.7.el5
    pam-0.99.6.2-12.el5
    glibc-2.5-107
    libselinux-1.33.4-5.7.el5
    audit-libs-1.8-2.el5
    fipscheck-lib-1.2.0-1.el5
    openssl-0.9.8e-22.el5_8.4
    glibc-2.5-107
    zlib-1.2.3-7.el5
    glibc-2.5-107
    glibc-2.5-107
    glibc-2.5-107
    krb5-libs-1.6.1-70.el5
    krb5-libs-1.6.1-70.el5
    krb5-libs-1.6.1-70.el5
    e2fsprogs-libs-1.39-35.el5
    nss-3.13.6-3.el5_9
    glibc-2.5-107
    libsepol-1.15.2-3.el5
    krb5-libs-1.6.1-70.el5
    keyutils-libs-1.2-1.el5
    nss-3.13.6-3.el5_9
    nspr-4.9.2-2.el5_9
    nspr-4.9.2-2.el5_9
    nspr-4.9.2-2.el5_9
    glibc-2.5-107
    glibc-2.5-107
    pam-0.99.6.2-12.el5
    file /lib64/security/pam_hulk.so is not owned by any package
    pam-0.99.6.2-12.el5
    cracklib-2.8.9-3.3
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    pam-0.99.6.2-12.el5
    Code:
    root@titanium [~]# rpm -qf /lib64/security/pam_hulk.so
    file /lib64/security/pam_hulk.so is not owned by any package

    Code:
    root@titanium [~]# strings /lib64/security/pam_hulk.so
    __gmon_start__
    _init
    _fini
    __cxa_finalize
    _Jv_RegisterClasses
    pam_hulk_version
    openlog
    vsyslog
    closelog
    pam_get_item
    readline
    fileno
    read
    feof
    get_hulk_key
    fopen
    fclose
    disconnect_hulk
    memset
    sigfillset
    sigaction
    alarm
    socket
    connect
    snprintf
    write
    tell_hulk
    hulk_enabled
    pam_sm_authenticate
    pam_get_user
    time
    pam_sm_setcred
    libc.so.6
    __xstat
    libpam.so.0
    _edata
    __bss_start
    _end
    LIBPAM_1.0
    GLIBC_2.2.5
    ATSubH
    /var
    /run
    /cph
    ulkd
    .socf
    cphulk_version_2.7
    PAM-hulk
    timeout while connecting to cphulkd
    /var/cpanel/cphulkd/keys/pam
    error getting hulk key
    error setting sigalrm
    opening socket
    failed to connect stream socket
    error reading welcome from stream socket
    Unexpected welcome messge from cphulkd: %s
    AUTH pam %s
    error reading auth status from stream socket
    error logging into hulkd: %s
    PAM_AUTHENTICATE system %s %s 1 %ld 0 0 %s
    PAM_SETCRED system %s %s 1 %ld 0 0 %s
    err writing to stream socket
    error reading PAM_? from stream socket
    Brute force detection active: %s
    /var/cpanel/hulkd/enabled
    /var/cpanel/cphulk_enable
    crond
    On a cPanel/Centos server with cPHulk Brute Force Protection disabled. And does not have the libkeyutils.so.1.9 file present.
    pam_hulk.so is installed regardless of it being enabled or not. that setting is enforced elsewhere in the stack (specifically by hulkd not running).

  11. #261
    Join Date
    Mar 2005
    Posts
    187
    @mattmackman

    Did you check the cron log?

    Attacker checked the cron log 3 times like he was waiting for something to happend in that specific hour...

    # cat /var/log/cron | egrep -i Feb 18 07

    What was it?

  12. #262
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by SoftDux View Post
    Ok, so it's probably safe to say the hacker now simply use another file name.... which also means all automated scans for the original file name will be void soon.


    Normal file, part of cpanel cpkhulkd bruteforce protection. It is NOT part of a rpm.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  13. #263
    Join Date
    Mar 2006
    Location
    Johannesburg,South Africa
    Posts
    601
    Quote Originally Posted by egillette View Post
    The question of the hour for sure.

    Let me know if I can be of assistance.

    In the interim, I whipped up a quick and dirty little bash script that others here or reading this thread may be able to use to remove the file from their affected machines.

    Instruction on how to use this script:

    1) Login to your server as root, or login to your server and use sudo or su to gain root access.
    Code:
    wget http://www.ericgillette.com/clients/exploit-cleanup
    sh exploit-cleanup
    2) Watch output from the script to tell you if the file is removed.

    Code:
    updatedb && locate libkeyutils.so.1.9
    3) No *output* from locate is an acceptable result.

    Disclaimer: This is a quick and dirty "for now" solution, if anyone has any input to add, feel free to do so -- specifically until we establish how the file ended up on the affected systems in the first place, there is the possibility it could somehow come back.

    Note: If you receive an error when running the final command, it's possible you don't have mlocate installed, in which case you should run the code below and then run the the code from Step 2:

    Code:
    yum install mlocate
    Can I add:

    If the server doesn't have the original /lib/libkeyutils.so.1.3 or /lib64/libkeyutils.so.1.3 (CentOS 5.9 has /lib/libkeyutils.so.1.2?) then SSH, FTP, wget, in fact anything that requires this login library will fail and the server owner won't be able to login to his server again. I haven't tested this with console access but remote access will be "broken". So your script may want to check for the original file first and probably re-install the keyutils-libs-1.2-1.el5 or (corresponding for the OS in question) library.
    South African Web Hosting - http://www.SoftDux.co.za || SA WebHostingTalk - http://www.webhostingtalk.co.za

  14. #264
    Join Date
    Oct 2011
    Location
    West Palm Beach, FL
    Posts
    52

    Exclamation

    Quote Originally Posted by TravisT-[SSS] View Post
    We can confirm that the pam module used by CPanel is not been exploited. We have the same strings match up on a clean box not being used to do spam nor have we seen any activity hinting to an exploit.

    I think it would be safe if we don't jump to conclusions here without posting verbose logs from snoopy ect.. to backup findings.

    On a sidenote, I think it would be a good idea if everyone was running snoopy till we find the root cause.

    On a random note: I think it is safe to say though we should expect the attackers to start randomizing the file at some point to attempt to remain more hidden or change their tactics up some.
    Hey Travis, so far our servers seam ok, no hacks, but I do have one server on CentOS 5.9 x64 that SSH wont respond, and wont restart, although the service shows as up and running in WHM under service status, any thoughts on that?

    Adding snoopy sounds like a good idea, any special instructions in setting it up, or is it an install and it works sort of thing?

    Also I wanted to thank the community of brilliant minds here working on this as well as the help of the cPanel, Cloud Linux and CSF staff, but my question is where the f*#k is Red hat on this? It seams like it has been narrowed down to their distro only and they are AWOL in these discussions.
    ██ Lead Dog Graphic Studio, LLC - Custom Designed Websites
    ██ Custom Web Design | Professional WordPress Design | Responsive Web Design
    ██ Come see why we are the Lead Dog when it comes to web design and awesome customer service.

  15. #265
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    1,029
    I cannot blame RedHat here, as I am pretty sure no one here licenses OS directly from them.
    CloudLinux & CentOS are not RHEL. So, we cannot really blame them for not acting.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  16. #266
    Join Date
    Oct 2011
    Location
    West Palm Beach, FL
    Posts
    52
    I didn't blame them, I thanked them for being here. I asked where was Red Hat in this, you are right, no one licenses from them, however they are the ones capable of plugging the hole, so yes they need to be involved.
    ██ Lead Dog Graphic Studio, LLC - Custom Designed Websites
    ██ Custom Web Design | Professional WordPress Design | Responsive Web Design
    ██ Come see why we are the Lead Dog when it comes to web design and awesome customer service.

  17. #267
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Quote Originally Posted by LeadDogGraphicStudio View Post
    I didn't blame them, I thanked them for being here. I asked where was Red Hat in this, you are right, no one licenses from them, however they are the ones capable of plugging the hole, so yes they need to be involved.
    The PTRACE flaw is probably unrelated to all of this, and as mentioned earlier, for that flaw to work there has to be ideal timing conditions within the kernel and it doesn't appear to be easily exploitable. Yes it's a flaw, but it doesn't look like a priority right now based on the degree of difficulty to make it work... so I kind of understand RedHat not dropping what they are doing to fix it.

  18. #268
    Join Date
    Jul 2007
    Location
    Florida
    Posts
    244
    Quote Originally Posted by brianoz View Post
    Steven - sorry for being unclear. I didn't mean to imply the initial attack was from ssh; what I meant was that an iptables block on ssh stopped them reconnecting, exactly as you've seen.

    Is the following consistent with what you've seen?

    1. User account compromised at PHP level
    2. Compromised account used to hack root and backdoor sshd via libkeyutils
    3. Spam sent

    The question being, how is the #2 root hack being done, #1 could be through any vulnerable site CMS etc.
    Confirmation(yea or nay) if all compromised systems were serving(not just installed) PHP?


  19. #269
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Quote Originally Posted by Ceetoe View Post
    Confirmation(yea or nay) if all compromised systems were serving(not just installed) PHP?
    Why does it matter? PHP alone isn't going to lead to a root compromise.

  20. #270
    Join Date
    Oct 2012
    Location
    Europe and USA
    Posts
    991
    Quote Originally Posted by richammond View Post
    There's an update for CSF which fixes this issue.
    See changelog for 5.76

    http://www.configserver.com/free/csf/changelog.txt

    =>Added new LF_EXPLOIT check SSHDSPAM to check for the existence of
    /lib64/libkeyutils.so.1.9 or /lib/libkeyutils.so.1.9
    Congratulations to the CSF team for the fast response, but CSF 5.76 caused us huge network issues, although our servers were absolutely clean and not infected by this exploit.
    It was blocking network connections from IP addresses not existing in the firewall rules, rendering our DNS cluster unusable, bringing down our billing and monitoring systems and blocking traffic from a lot of visitors.

    Fortunately we had kept the installation files of version 5.71 so we uninstalled the new version 5.76 and reinstalled 5.71.

    Sorry but we will not use CSF for scanning the exploit discussed in this topic.
    Rendering an entire network of servers unreachable to 50% of the world, to protect from an exploit is not a solution.

    Thanks
    Last edited by NetworkPanda; 02-18-2013 at 06:03 PM.
    NetworkPanda :: Web Hosting SSD Powered :: Reseller Hosting
    Instant activation, fast servers, NVMe SSD disks, cPanel, Softaculous 1-click apps installer, daily backups
    Multiple hosting locations: USA, Canada, France, UK, Germany, Italy, Spain, Poland, Finland

  21. #271
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Quote Originally Posted by NetworkPanda View Post
    Congratulations to the CSF team for the fast response, but CSF 5.76 caused as huge network issues.
    It was blocking network connections from IP addresses not existing in the firewall rules, rendering our DNS cluster unusable and blocking traffic from a lot of visitors.

    Fortunately we had kept the installation files of version 5.71 so we reverted to it.

    Sorry but I will not use CSF for scanning the exploit discussed in this topic.
    Rendering an entire server unreachable to 50% of the world, to protect from an exploit is not a solution.

    Thanks
    They released a new one that appears to have fixed it.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  22. #272
    Join Date
    Jul 2007
    Location
    Florida
    Posts
    244
    Quote Originally Posted by Patrick View Post
    Why does it matter? PHP alone isn't going to lead to a root compromise.
    It may not matter. Are you vouching for the correct configuration on all compromised servers? I'm certainly not just as I would never vouch that there are no vulnerabilities with installed packages. At this stage we just don't know.


  23. #273
    Join Date
    Oct 2012
    Location
    Europe and USA
    Posts
    991
    Quote Originally Posted by Steven View Post
    They released a new one that appears to have fixed it.
    After what happened, we will not take the risk with any new CSF version, we will keep 5.71 which has been working fine for us for ages.

    Let's hope their CXS malware scan tool gets updated soon to scan for the exploit, as we are also using it and it is proved to be better than CSF when used for malware detection.
    Last edited by NetworkPanda; 02-18-2013 at 06:11 PM.
    NetworkPanda :: Web Hosting SSD Powered :: Reseller Hosting
    Instant activation, fast servers, NVMe SSD disks, cPanel, Softaculous 1-click apps installer, daily backups
    Multiple hosting locations: USA, Canada, France, UK, Germany, Italy, Spain, Poland, Finland

  24. #274
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    Quote Originally Posted by Ceetoe View Post
    It may not matter. Are you vouching for the correct configuration on all compromised servers? I'm certainly not just as I would never vouch that there are no vulnerabilities with installed packages. At this stage we just don't know.
    It doesn't matter, no vulnerability in PHP alone is going to lead to a root compromise therefore making it 100% irrelevant.

  25. #275
    Join Date
    Nov 2002
    Location
    Oklahoma
    Posts
    702
    Quote Originally Posted by NetworkPanda View Post
    We will not take the risk with any new CSF version, we will keep 5.71 which has been working fine for us for ages.

    Let's hope their CXS malware scan tool gets updated soon to scan for the exploit, as we are also using it and it is better than a firewall at scanning for malware.
    Your issue with CSF had nothing to do with the malware scanning portion of it, it just happened to be included in the same update. The problem was the attempt at switching to conntrack which has since been effectively reverted in 5.78. Obviously it would have been better had this not occurred but refusing to update going forward is a bit much. If anything, disable automatic updates and test them in a pre-production environment before deploying them.
    Dathorn, Inc. - Premium cPanel/WHM Hosting since 2002! Check Out Our Blog!
    Experience the Dathorn Difference! - andrew@dathorn.com
    LiteSpeed | Clustered DNS | CloudLinux | CageFS | KernelCare | Imunify360
    Pure SSD Storage | Off-Server & Off-site Backups | Softaculous | SpamExperts

Page 11 of 61 FirstFirst ... 89101112131421 ... LastLast

Similar Threads

  1. ****`it Rootkit, Tuxtendo Rootkit
    By ISpy in forum Hosting Security and Technology
    Replies: 4
    Last Post: 06-22-2010, 11:27 AM
  2. Which server builds are you rolling out?
    By GeekMe in forum Dedicated Server
    Replies: 11
    Last Post: 04-18-2010, 08:03 AM
  3. Getting the ball rolling ...
    By policefreq in forum New Members
    Replies: 1
    Last Post: 08-19-2006, 11:16 PM
  4. Getting company to get rolling
    By Overclocked in forum Running a Web Hosting Business
    Replies: 19
    Last Post: 08-03-2004, 04:02 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •