Results 1 to 29 of 29
  1. #1

    problems at command line

    I am running cpanel/RHEL and I got a page that my cpu was overloaded. I SSH'd in and tried to run top. However, I got the following error:

    [/scripts]# top
    top: error while loading shared libraries: libncurses.so.4: cannot open shared object file: No such file or directory
    [/scripts]#

    Also, when I try to do an ls on a file, I get the following:

    [/scripts]# ls a* -l
    /bin/ls: unrecognized prefix: do
    /bin/ls: unparsable value for LS_COLORS environment variable
    -rwx------ 1 root root 3222 Oct 21 01:45 adddns*
    -rwx------ 1 root root 3444 Nov 21 2002 addfpmail*
    -rwx------ 1 root root 3270 Nov 21 2002 addfpmail2*
    -rwx------ 1 root root 231 Apr 29 2002 addnetmaskips*
    -rwx------ 1 root root 496 Sep 5 2002 addnobodygrp*
    -rwx------ 1 root root 1463 Apr 22 2002 addpop*
    -rwx------ 1 root root 1599 May 11 2001 addservlets*
    -rwx------ 1 root root 865 Mar 10 2002 addstatus*
    -rwx------ 1 root root 164 Nov 8 2002 adduser*
    -rwx------ 1 root root 2270 May 11 2001 adduser.old*
    -rwx------ 1 root root 315 Jan 23 2002 admin*
    -rwx------ 1 root root 10097 May 11 2001 admin.old*
    -rw-r--r-- 1 root root 654 Aug 25 2003 anonuser.patch
    -rwx------ 1 root root 5720 May 27 2002 antirelayd.shar*
    -rwx------ 1 root root 1847 Jun 29 2003 apachelimits*
    [email protected] [/scripts]#

    Any idea what is wrong here???
      0 Not allowed!

  2. #2
    smells like someone broke into machine and installed rootkit.

    regards,
    M.
    Powered by AMD & FreeBSD.
    "Documentation is like sex:
    when it is good, it is very, very good;
    and when it is bad, it is better than nothing."
      0 Not allowed!

  3. #3
    Can you please expand on that? I have no idea what that means.
      0 Not allowed!

  4. #4
    Join Date
    Apr 2003
    Location
    San Francisco, CA
    Posts
    428
    did you set up any aliases in your .bashrc file? It does seem fishy...

    /bin/ls: unrecognized prefix: do
    /bin/ls: unparsable value for LS_COLORS environment variable

    The environment variable thing, and the do command (syggesting a shell scriptite thingy going on) makes me think something is up with your shell...
    Hostito, Inc. - Web Hosting & Reseller Plans
    http://www.hostito.com
    [email protected] - 1 888 467 8486
      0 Not allowed!

  5. #5
    Originally posted by rrsnider
    Can you please expand on that? I have no idea what that means.
    there was a link recently to debian's mailing list. I think it is tr0n rootkit
    anyway, check with chkrootkit

    regards,
    M.
    Powered by AMD & FreeBSD.
    "Documentation is like sex:
    when it is good, it is very, very good;
    and when it is bad, it is better than nothing."
      0 Not allowed!

  6. #6
    Defintly a root kit.... probably because of the cpanel exploit

    do a search... there are a few informative posts from the weekend.

    My server is currently having the OS reloaded due to the same issue
      0 Not allowed!

  7. #7
    How do you run chkrootkit? When I type chkrootkit, I get the following message:

    -bash: chkrootkit: command not found
      0 Not allowed!

  8. #8
    Join Date
    Dec 2002
    Posts
    265
    You can download it and get more information from http://www.chkrootkit.org/
      0 Not allowed!

  9. #9
    You have the t0rn v8 rootkit. Checking it is probably not going to help since it just tells you you are rooted.

    From the sounds of it, you seem relatively inexperienced at using SSH. You should probably hire a system admin, or order a reinstall of your OS
    Like us on Facebook to qualify for discounts!
    http://www.sprintserve.net
    Offering: | Internap FCP Bandwidth! | Rebootless Kernel Updates! | Magento Optimized Hosting | Wordpress Hosting |
    Services: | Managed Multiple Cores 64bit Servers | Server Management |
      0 Not allowed!

  10. #10
    Join Date
    Dec 2003
    Location
    Mentor, Ohio
    Posts
    38
    i agree with sprintserve, and make sure your system is locked down next time..
    Compaq Linux Forums!
    http://www.cpqforums.com
      0 Not allowed!

  11. #11
    Originally posted by nightwar
    i agree with sprintserve, and make sure your system is locked down next time..
    Unfortunetly, cpanel is mostly to blame for this one
      0 Not allowed!

  12. #12
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Originally posted by BootsSiR
    Unfortunetly, cpanel is mostly to blame for this one
    No, cpanel's not to blame, the ones to blame are the individuals that don't secure their server.

    Granted, CPanel screwed up, but in this case, the warning has been out there for a few days and the protective measures to handle it as well. If someone doesn't follow through on those, that's not the fault of the software vendor.

    I'd agree with everyone else though, you've most likely been rooted. Run chkrootkit, see if it's been rooted, get backups and have the server formatted and reinstalled.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details
      0 Not allowed!

  13. #13
    Join Date
    Dec 2003
    Location
    Mentor, Ohio
    Posts
    38
    ya, i seen all the info over the weekend.. kinda why i hate control panels! none the less, CP's have taken this business over.. i still do things the old fashioned way without a CP

    anyhow.. if an exploit is found, and makes it way around quickly enough, theres not much to prevent this from happening!

    good luck!
    Compaq Linux Forums!
    http://www.cpqforums.com
      0 Not allowed!

  14. #14
    You have to check your systen whether chkrootkit is installed(Use locate chkrootkit) and if there's nothing, then it was not installed and you have to see Westech's post and the url to download and install it.

    After you have done the install, cd to the directory which is usually in /usr/local/src/chkrootkit-0.43/ then issue the command ./chkrootkit and it will detect your system's affected(infected) system/files.

    Good luck
    WHO AM I? CLICK HERE!
      0 Not allowed!

  15. #15
    I ran chkrootkit and here is what is infected:

    Checking `ifconfig'... INFECTED
    Checking `login'... INFECTED
    Checking `pstree'... INFECTED
    Searching for t0rn's v8 defaults... Possible t0rn v8 \(or variation\) rootkit installed
    Searching for Showtee... Warning: Possible Showtee Rootkit installed
    Checking `bindshell'... INFECTED (PORTS: 465)
    Checking `lkm'... You have 3 process hidden for ps command
    Warning: Possible LKM Trojan installed

    Is there anyway to fix the problem(s) without a restore? I saw some posts on this but there does not seem to be a consensus on whether to restore or repair.
      0 Not allowed!

  16. #16
    No, cpanel's not to blame, the ones to blame are the individuals that don't secure their server.

    Granted, CPanel screwed up, but in this case, the warning has been out there for a few days and the protective measures to handle it as well. If someone doesn't follow through on those, that's not the fault of the software vendor.

    I'd agree with everyone else though, you've most likely been rooted. Run chkrootkit, see if it's been rooted, get backups and have the server formatted and reinstalled.
    I have to disagree with this. I had been following here at WHT and at Cpanel and updated our servers when told to do so. We still had one server get hit with it. Fortunately there were other measures in place that prevented a lot of damage but it still happened. In the short amount of time between the release of the exploit and the patch that cpanel released, it still a happened.
      0 Not allowed!

  17. #17
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Originally posted by rrsnider

    Is there anyway to fix the problem(s) without a restore? I saw some posts on this but there does not seem to be a consensus on whether to restore or repair.
    Honestly, NO, there isn't. Now, these CAN be removed, rather easily, however you'll still most likely have the stuff that they put in there that WASN'T detected, and you'll end up losing sleep over whether or not your system is still infected.

    The best route to take:
    A> Have a new system setup (or have this one tossed and formatted).
    B> Have your users change their passwords (this is CRITICAL, because once in, the user can run software to detect their password, and I'd bet anything that just a few of them had dictionary passwords).
    C> Restore accounts .
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details
      0 Not allowed!

  18. #18
    What exactly is t0rn v8 going to allow my hacker to do to my server?
      0 Not allowed!

  19. #19
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Well, you've got a lot more than t0rn in there, though that's just the start of things. Your basic system binaries have been corrupted (look at the output from chkrootkit and you'll see which ones).

    In short:
    This user (whoever they are) has complete access to your server by now. It's very safe to assume this, because they've already modified login and installed their own kits on your server. There's nothing they CAN'T do beyond this point.

    Now, most of these are kiddies, running a binary, that's it, not knowing what they're doing past some shell script or binary, but that's most definitely not a chance worth taking here.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details
      0 Not allowed!

  20. #20
    Basically it gives the hacker root (ie the highest possible level) access to your server so that he/she can do anything they want.

    The most commonly occuring two things (I think) are the following:

    Using your server to participate in a dDOS (Distributed Denial of Service), ie flooding a remote person with loads of traffic, using up your bandwidth

    Using your server to send spam email through, causing your ip addresses to get blacklisted and your server to slow down horribly under the tide of email processes (the fact you said your server was overloaded indicates this might already be going on).

    So ultimately, however annoying it is and however much of a hassle it is you HAVE TO reformat the server and start again (after taking the relevant backups of peoples sites of course). I would also suggest that once you have got your server back up you get someone to properly secure your server for you (I think there are a few people who post on WHT that do it).
      0 Not allowed!

  21. #21
    How can I check if they are sending spam from my server?
      0 Not allowed!

  22. #22
    Unfortunately you can't check very easily as the entire point of the rootkit is that it replaces all the commands that would show you the processes sending the spam with ones that will deliberately not show you, therefore you almost certainly will not be able to see what they are doing.

    However, in this case it appears the rootkit is incompatible with certain libraries on your system which is why the commands are failing.

    The only sure fire way (I think) to know if somebody is sending spam from your server now is to stick the machine on a hub (NOT a switch) with another machine running a packet sniffer and see what is going in/out from it.

    However, as you might not have physical access another option is to try searching for your IP on block lists (which list sources identified as sending spam), however, your IP may not have yet been added to these lists even if it is sending out spam so this isn't that good an option.
      0 Not allowed!

  23. #23
    Join Date
    Apr 2002
    Location
    Philly Pa
    Posts
    130
    What does it do for the script kiddie? It gives him/her your machine to do as he/she wishes to do with it.

    Even though i hate cpanel, and cant stand what it does for us hard working admins. I have to admit its not 100% their fault. the bugs were released before the coder was notified. This is against ethics. There is standards to releasing bugs.

    1. Notify Programmer.
    Alert programmer your giving him so so time to release a patch before you release the advistory.
    2. release the notification and include the fix.

    This clown just started post all over. Servers were getting hijacked before the fixes were even released. Hell, servers were getting hijacked before the programmer even knew.

    But it is everyones right to secure their own machines. You install 3rd party programs to make your life easier, your asking for it.

    Dont be lazy. Be old fasion and secure. If you dont know much about unix/linux and want to start a hosting company, please hire a fulltime admin. Your making the life of real administrators a pain in the rear.


    1. I'm tired of my machines getting port scaned by hijacked machines.
    2. I'm tired of spam from hijacked servers.


    Any other admins want to add to this list?

    3. ????
      0 Not allowed!

  24. #24
    I've heard enough. I am going to start the restore process ASAP. I have daily/weekly backups of my sites (using Cpanel backup) and I will restore the sites from last week (prior to rootkit). Here are my questions:

    1) Anything I should take a backup of before I do the restore?
    2) Anything special I need to do to restore the sites? I use the cpanel backup method to a SAMBA drive.
    3) Any directories I should avoid restoring?
    4) I saw that I should change all passwords. Any other advice?

    Thanks.
      0 Not allowed!

  25. #25
    Join Date
    Apr 2002
    Location
    Philly Pa
    Posts
    130
    you want to get rid of it all. dont save anything.
      0 Not allowed!

  26. #26
    Join Date
    Dec 2003
    Location
    Mentor, Ohio
    Posts
    38
    You know what, this comment:

    Dont be lazy. Be old fasion and secure. If you dont know much about unix/linux and want to start a hosting company, please hire a fulltime admin. Your making the life of real administrators a pain in the rear.


    by the person up there, as much i have i bite my tounge reading these posts, i have to agree... most resellers do not know what httpd.conf even is!!! .... at any rate.. im having a bad day.. sorry!
    Compaq Linux Forums!
    http://www.cpqforums.com
      0 Not allowed!

  27. #27
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Getting rid of everything while advisable isn't going to be a realistic solution. In fact, it's just not possible to do that AND keep your customers.

    As far as what to keep? Don't, restore your backups from the backups you have and PRAY that they weren't infected. If they were, you'll be going through this again shortly.


    I agree, if you don't know what you're doing, find a sysadmin. It's a bit more expensive, but in the long run it pays off, if the admin knows what he (or she) is doing. There really is no excuse for not securing your server.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details
      0 Not allowed!

  28. #28
    First of all, thank you to everyone for the responses, they really helped. Secondly, I am a professional software developer that has gotten into web design/hosting as a hobby. I am well versed in many operating systems other than Unix. I find these forums useful in gathering information and solving problems. Maybe I should do things the "old fashion way" and spend 3 months to Master Unix/Linux. Anyways, I guess we were all inexperienced at this at one time or another.

    Again thanks for the help.
      0 Not allowed!

  29. #29
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Originally posted by rrsnider
    Maybe I should do things the "old fashion way" and spend 3 months to Master Unix/Linux.

    3 months? Try years. Hell, I've been working with Linux/Unix for (almost) 10 years now and I STILL don't have it completely mastered.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details
      0 Not allowed!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •