You've got a few websites, you've got a VPS, you know what Linux is but... you've been hacked. These hackers have set up some
sorta crazy automated way of sending out over 100,000 spam mails (literally) and you are stuck. Your hosting provider sent
you an e-mail giving you a little bit of information about whats going on but you're being pressed to fix a problem you know
nothing about and you're ready to pull your hair out.
Well worry not, with this quick lil guide you should be able to track down the issue and resolve the problem within a few
minutes to an hour. I'll have made a few assumptions of the folks reading this. You have a linux server that you are able to
gain access to via ssh and you just so happen to know a few commands to get you around the system. This guide is only
going to be for tracking down spam mail being sent out via Exim on a cPanel server. I'll update it eventually with information
for postfix and qmail.
On we plow...
Ok so let's see how bad the situation is here. Time to check the mail queue and see how backed up it is.
YUP! I'd say that's a few... time to see what all the fuss is about. Now this command will display what is
in the queue. With such a large amount it will literally FLY past your screen faster than a Porsche 911 GT3.
Feel free to pipe over to more so you can get a handle on things, or just get happy with the ^C.
So we got something to work with now. See the third column there? The one with the mail ids... example: 1W3DDO-0002xo-H7. We're
going to use that to check the header and hopefully get a glimpse at where this is coming from.
So by looking at the header we can gather a few things that will help us. The userid and lucky for us, the script sending the spam! Don't worry, we're not always so lucky.
A lot of times this is missing and we gotta go hunting. So I'll show you how to do that anyway, for a good reason too! We won't pay much attention to the userid now as it's
obvious that this is coming from a script. If you're confused it's the following line that gives it away:
As you can see it shows us the script that is sending the mail. Now if you know your way around your server you'll be able to navigate to the directory where fubar.fu points
to and get that nasty script!
But if you don't just follow along, we'll get to that.
Ok so let us just pretend that the X-PHP-Script location wasn't there for us. What do? Well... you could dig through the exim_mainlog looking for clues or you can run this nifty
one liner that will show you just where scripts are hiding!
Now we can ignore the first line there because obviously exim is going to be sending a large amount of mail. We can ignore the csf line too... cause well... yup it sends a lot of
mail as well. But what about those in between you ask? BINGO! There's our lucky charms... sorta. Let's get in the directories and see what's in the pot of gold.
-bash-4.1# cd /home/fubar/public_html/includes/phpmailer
-bash-4.1# ls -al
drwxr-xr-x 2 fubar fubar 4096 Jan 14 08:27 .
drwxr-xr-x 8 fubar fubar 4096 Jan 14 12:02 ..
-rw-r--r-- 1 fubar fubar 224 Jan 16 2011 class.phpmailer.php
-rw-r--r-- 1 fubar fubar 214 Jan 16 2011 class.smtp.php
-rw-r--r-- 1 fubar fubar 44 Jan 16 2011 index.html
-rw-r--r-- 1 fubar fubar 26421 Jan 16 2011 LICENSE
-rw-r--r-- 1 fubar fubar 7373 Jan 14 08:27 wishlistcBU.php
Notice anything out of the ordinary? Besides the script that was pointed out in X-PHP-Script... wait, remember now that doesn't exist so we have no idea! But from a glance we can see
that everything is from Jan 16th 2011 EXCEPT.... wishlistcBU.php! We just gotta check it out and see if it's legit or not though.
WOW NOW... what's all that stuff you say? Well... it's bad stuff. It's the dirty script that is responsible for plaguing the internet with viagra ads and promises of
fame and fortune from some Nigerian prince. Now is your chance to fight back! Lets get that bad boy... but first when in doubt just rename and disable it. This way if
you just so happen to make a mistake, you can always undo it easily. (Depending on your php handler sometimes chmod'ing the file to 000 will suffice)
-bash-4.1# ls -al | grep guest
---------- 1 fubar fubar 7373 Jan 14 12:52 guestbookP89.bak
Are we noticing a pattern here yet? Good So just continue doing what we've been doing until we are able to satisfy the results that our magic oneliner gave us.
The last one should be easy to figure out, so I'll just paste the aftermath in the top portion of the next example. In that example since we've "cleaned up" the
spam scripts we'll clean out the queue and see how things look afterwards.
Now lookie there, exim is reporting 0 in the queue. Yeah... it was right after erasing the entire thing but hey... I bet it stays! Now DO NOT use the rm -rf command
if you are not familiar with what it does. Also do not just rm -rf /var/spool/exim if you have mail in your queue that isn't malicious or that you care about. If you
don't care about it and just want it all gone and are familiar with the consequences of using the rm -rf command, then give it a shot.
Now if you are concerned then there is one way we can remove the spam mail from the queue and keep the good stuff. We'll add the [email protected] e-mail address
we saw sending the spam from our investigation thus far.
You'll see a nice list scroll by of all the mailids being removed from the queue. And just like that... you've won! You conquered the spam! Give yourself a pat on the back,
a pat well earned!
But wait... THERE'S MORE! How did this crap get here?
Should we break out the virus scanner? Sure it can help... but not with finding the vulnerability that allowed the content to get here. Soooooo what do we do? Well we use
the best tools in our arsenal... logs and time-stamps of course!
So lets grab pick a file and see what we can find. I pick the time-stamp for the spam script formally known as guestbookP89.php
Alrighty we got something, weeee! Well I see guestbookP89.php but I also see a POST to some other weird looking script. For sake of simplicity I'm not going to cat this file (but you should). Let's
go ahead and disable that file and match it's time stamp up with the logs too. Don't forget that sometimes you'll need to rename the file as well.
-bash-4.1# cd /home/fubar/public_html/images/stories/
-bash-4.1# chmod 0 wdt2998g.php && pwd && ls -al
---------- 1 fubar fubar 34452 Jan 14 03:39 wdt2998g.php
Hey now... this is something different. So we can see wdt2998g.php but what is going on right ahead of it? Well it does take an eye to notice but luckily the exploitation did leave BOT for JCE
tacked all over the place so we can safely assume it has something to do with the above lines. That and the matching ip address (note: the ip addresses will not always match).
From this we can gather that com_jce was utilized somehow to assist in the upload of wdt2998g.php. Well as life would have it older versions of Joomla are actually vulnerable to com_jce exploitation.
This is exactly what you see here. The solution? Upgrade your components... actually upgrade all the things. Plugins, themes, modules, all the fancy stuff you spent hours downloading and forgetting
about too! Upgrade what you use and remove what you don't.
Now wait a sec... I did say something about older versions of Joomla right? Let's check the version we're using. We did just get exploited due to an outdated com_jce that is associated with outdated
Joomla installs. We'll go into the directory that we have Joomla installed and run another magic one liner!
-bash-4.1# cd /home/fubar/public_html/
-bash-4.1# grep -r --include=*version.php "RELEASE[[:space:]]*=\|DEV_LEVEL[[:space:]]*=" *
libraries/joomla/version.php: var $RELEASE = '1.5';
libraries/joomla/version.php: var $DEV_LEVEL = '23';
Look at that! Not only are we outdated... but our version isn't even supported anymore! It's reached end of life and who knows what else can be done to us. This is basically just turning your VPS into
a sitting duck.
So it's time to remove (after you have confirmed the files are in fact malicious and not needed) the malicious files we disabled and renamed. Then it's time to do a nice big ol' upgrade on Joomla.
Now... NOW the spam has been defeated. WOOT!
I hope you have learned something from this little blurb (lol). I will be adding information for WordPress exploitations as well as how to tell the difference between spam that is being sent by a script
and when it's being sent from a compromised account that someone or something is authenticating to. Keep in mind that there are are a ton of ways to skin a cat and that this is just a quick and dirty
way to try and tackle common spam infestations. This information is not meant to be a cure-all and I am not responsible for any damages that may occur because of it's use.
One thing you could also do is rate limit outgoing SMTP connections with iptables to 1/minute or so. This way, even if a spammer gets access in some form, they can only send very slowly and will likely give up before it becomes a problem.
Would be interesting to see a similar guide for other mail servers though. I'd be very interested in the postfix and qmail ones.
I wanted to chime in also and say excellent article, but wanted to stress what Bazfrank mentioned above "One thing you could also do is rate limit outgoing SMTP connections with iptables to 1/minute or so."
This won't help you track down spam, but it's a measure that can help slow things down incase you get to the issue late.
Webhostpython.com - Reliable Shared, Reseller, and KVM VPS Hosting Services.
Dual Octa Core Xeon E5 Servers. RAID10 Storage. Enterprise DDOS Protection. Pure SSD Plans
24/7 Support | Live Chat | In-House Support Staff | 1-800-929-9061 | Dallas, TX