Results 1 to 25 of 27
-
03-24-2005, 08:05 PM #1Newbie
- Join Date
- Apr 2004
- Posts
- 22
Our server has been compromised - I am crying for help
Dear webmasters !
I hope you are in the best of your spiritual and physical health.
I have a Red hat linux server from http://www.servermatrix.com. The load on our server is very high these days and our online forum (8500 + members) goes offline a number of times everyday. This is resulting in thousands of visitors turning away. We checked the server status and found that disk sda3/(var) is 100% full. Servermatrix.com opened a ticket to tell us that there are some malicious files and a trojan horse in /tmp. When we checked "Manage Mail queue", we found that there are more than 1 lac messages. These emails are sent to nobody@mohammadirfan (mohammadirfan is our server name).
I dont have much technical knowledge and I cannot do much. My friend used to manage this server for us but he is on holidays. I cannot sleep these days. One question comes in my mind that is the hacker or spammer reading the emails sent from/to our server or he is just sitting in /tmp with his malicious files. I am crying for help. What should I do?
-
03-24-2005, 08:08 PM #2Retired Moderator
- Join Date
- Mar 2004
- Location
- Singapore
- Posts
- 6,990
Check if linuxguy is online, try to PM him, he may be able to help you.
-
03-24-2005, 08:12 PM #3Web Hosting Master
- Join Date
- Aug 2002
- Location
- London, UK
- Posts
- 9,039
Get yourself an admin to look over the server for you...
www.ezsm.netMatt Wallis
United Communications Limited
High Performance Shared & Reseller | Managed VPS Cloud | Managed Dedicated
UK www.unitedhosting.co.uk | US www.unitedhosting.com | Since 1998.
-
03-24-2005, 09:23 PM #4Web Hosting Master
- Join Date
- Mar 2005
- Posts
- 1,073
Have you checked the log files?
You may want to check the running processes on your server as well, if you have diffuculty understanding what most of it means post the log here please
-
03-24-2005, 10:29 PM #5Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
First off i would check if the box was root compromised. run rkhunter ot chkrootkit, if it was. Its time for a osreinstall. If its not, start looking though the domlogs for wget, etc and see if you can find some exploitable php scripts. thats a start.
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
03-25-2005, 07:11 AM #6Junior Guru
- Join Date
- Nov 2002
- Location
- The Netherlands
- Posts
- 222
Is your server running the latest kernel/software and up-to-date?
Have you checked what kind of malicious files are in the /tmp?
-
03-25-2005, 02:05 PM #7Web Hosting Guru
- Join Date
- Jan 2005
- Posts
- 268
Originally posted by thelinuxguy
First off i would check if the box was root compromised. run rkhunter ot chkrootkit, if it was. Its time for a osreinstall. If its not, start looking though the domlogs for wget, etc and see if you can find some exploitable php scripts. thats a start.
what is the command for looking though the domlogs for wget and etc... Thanks
-
03-25-2005, 03:40 PM #8Web Hosting Master
- Join Date
- Jan 2002
- Posts
- 1,053
What security precautions did you take when setting up your server? Did you setup some of the most basic things such as a firewall? Change the permissions of wget and other alike functions?
-
03-25-2005, 04:16 PM #9Junior Guru Wannabe
- Join Date
- Mar 2005
- Posts
- 86
Basic View:
more /path/to/log/logfile | grep wget
man more and grep for more switches
-
03-25-2005, 07:02 PM #10
Just because your server's got stuff in /tmp doesn't mean it's been rooted, but chances are you've got something goin on there. Start with the malicious stuff in /tmp, try to find out where it came from. Here's how, and usually this works.
Let's say you have a suspicious file in /tmp called r00tedoor (or whatever).
The first thing you want to do is verify your ps binary (/bin/ps usually). If ls -la /bin/ps returns something like today, or some odd date, then you'll need to get it replaced correctly.
The second thing you want to do find out if it's RUNNING (ps -xua | grep r00tedoor).
IF it's running, try to cd /proc/procnum , then do an ls -la . Sometimes you'll get more info out of that, like where it's residing, etc (though in this case you know it's /tmp).
Then, you want to kill the process (note: kill it only after you have the obvious information from it).
Once you've killed the process, remove the file, pretty simple there.
Now, where did it come from?
Most likely, this came from apache, or someone's vulnerable script. So, you need to go through and find out which one. To do this, go into your logs directory (/usr/local/apache for CPanel, /etc/httpd/logs for DirectAdmin) and just type the following:
grep -R r00tedoor | more
most likely you will come up with at least one instance of this. Then you can deal with the customer with the outdated/vulnerable script to ensure it doesn't come back in.Tom Whiting, WHMCS Guru extraordinaire
Linux problems? WHMCS Problems? Give me a shout
Check out my WHMCS Addons
-
03-25-2005, 09:52 PM #11WHT Addict
- Join Date
- Dec 2004
- Posts
- 120
Re: Our server has been compromised - I am crying for help
Originally posted by International Player
Dear webmasters !
I hope you are in the best of your spiritual and physical health.
I have a Red hat linux server from http://www.servermatrix.com. The load on our server is very high these days and our online forum (8500 + members) goes offline a number of times everyday. This is resulting in thousands of visitors turning away. We checked the server status and found that disk sda3/(var) is 100% full. Servermatrix.com opened a ticket to tell us that there are some malicious files and a trojan horse in /tmp. When we checked "Manage Mail queue", we found that there are more than 1 lac messages. These emails are sent to nobody@mohammadirfan (mohammadirfan is our server name).
I dont have much technical knowledge and I cannot do much. My friend used to manage this server for us but he is on holidays. I cannot sleep these days. One question comes in my mind that is the hacker or spammer reading the emails sent from/to our server or he is just sitting in /tmp with his malicious files. I am crying for help. What should I do?
Normal files would be: cpanel.tmp, sess* *.session*, mysql.sock etc.
Look for suspicious files running with ps aux or top and kill them asap.
After killing them delete them and secure your TMP partition.
Go to this link for a great tutorial on how to secure your /tmp partition.
Secure TMP Partition by eth0
-
04-27-2005, 02:45 PM #12Web Hosting Master
- Join Date
- Apr 2003
- Location
- NC
- Posts
- 3,093
Just securing your /tmp will not help against perl files since they are not run directly from the tmp but though perl.
I would suggest installing a firewall like apf and having both inbound and outbound monitored.
I would also suggest mod_security to be installed to help prevent stuff getting on your server in the first place.
There are guides for both on my website posted above.
There is a lot more that you can do and should do to secure your server. I would move the files from your tmp but place them someplace else so that somebody else can later analyize the files, like your sys admin.
ooops sorry about the bump, I did not check the month stampLast edited by eth00; 04-27-2005 at 03:00 PM.
John W, CISSP, C|EH
MS Information Security and Assurance
ITEagleEye.com - Server Administration and Security
Yawig.com - Managed VPS and Dedicated Servers with VIP Service
-
04-28-2005, 12:31 AM #13Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Just securing your /tmp will not help against perl files since they are not run directly from the tmp but though perl.Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
04-28-2005, 01:17 AM #14WHT Addict
- Join Date
- Nov 2002
- Posts
- 161
What kind of script are you using for you forum.
We had the same problem a while back and it turned out that there were a few outdated version of PHPBB on our server which were exploited.
Make sure there are no unsecured php or cgi script runing on your machine.
-
04-28-2005, 01:33 AM #15WHT Addict
- Join Date
- Dec 2003
- Posts
- 134
Originally posted by thelinuxguy
Even executing perl scripts can be stopped
I suppose by making PERL only run through the web?.
-
04-28-2005, 02:00 AM #16Junior Guru
- Join Date
- Oct 2003
- Location
- Long Island, New York
- Posts
- 220
I'd like to know the answer to this as well...
TWSites.com - Business Web Hosting Solutions & Server Management Since 2003
-
04-28-2005, 03:35 AM #17Aspiring Evangelist
- Join Date
- Jan 2004
- Posts
- 370
Originally posted by DoCk
I'd like to know the answer to this as well...
-
04-28-2005, 09:34 AM #18Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Nothing to do with apache etc, it has to do with the operating system. Papi get me in irc and I'll let you know later tonight.
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
04-28-2005, 09:39 AM #19Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Originally posted by mp3LM
Yeah, how so?
I suppose by making PERL only run through the web?
doing so can cause very large problems with the operating systemSteven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
04-28-2005, 09:52 AM #20WHT Addict
- Join Date
- Dec 2003
- Posts
- 134
Originally posted by thelinuxguy
doing so can cause very large problems with the operating system
So...what's the right way?
Would it have to do with a kernel recompile? (If you don't tell me, I'll just keep posting guesses =P).
-
04-28-2005, 10:00 AM #21Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Well, normally the kernel supports the functionality. I'll release the info and how to do it correctly once i have more testing.
Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance
-
04-28-2005, 10:04 PM #22Disabled
- Join Date
- Oct 2004
- Posts
- 250
and Steve.. our wait continues.. eehehe
guys, if you want the answer, you should pay, he has a business to run also.
-
05-06-2005, 03:34 PM #23Junior Guru
- Join Date
- Apr 2005
- Location
- Sweden
- Posts
- 241
I guess one way to do it could be to replace /usr/bin/perl (or whatever) with a wrapper script that checks first if the script is on a mountpoint that is noexec. If not, run the perl process (that you renamed something else) normally. If it is on a noexec mp, then return some kind of error.
Kinda ugly, but could work. Only needs to come up with a lightweight wrapper script that doesn't mess things up too bad. Anyone have one in store?
Sorry 'bout the bump...
-
05-06-2005, 04:23 PM #24Web Hosting Master
- Join Date
- Mar 2004
- Posts
- 1,007
Why dont hire someone to investigate/secure your server?
Instead of waitinga nd waiting you could hire someone like www.totalserversolutions.com
At least i have had a positive expirience with them when i needed help...Best Regards,
Namesniper
-
05-06-2005, 06:12 PM #25Problem Solver
- Join Date
- Mar 2003
- Location
- California USA
- Posts
- 13,681
Originally posted by The_Overl
I guess one way to do it could be to replace /usr/bin/perl (or whatever) with a wrapper script that checks first if the script is on a mountpoint that is noexec. If not, run the perl process (that you renamed something else) normally. If it is on a noexec mp, then return some kind of error.
Kinda ugly, but could work. Only needs to come up with a lightweight wrapper script that doesn't mess things up too bad. Anyone have one in store?
Sorry 'bout the bump...Steven Ciaburri | Industry's Best Server Management - Rack911.com
Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance