Results 1 to 11 of 11
  1. #1

    Help my server is sending spam and my abuse department is mad at me! (A HOWTO)

    I decided to write up a quick and dirty read for those interested in tracking down spam on their VPS. I'll add to this occasionally to keep it interesting. For now... ENJOY


    You've got a few websites, you've got a VPS, you know what Linux is but... you've been hacked. These hackers have set up some
    sorta crazy automated way of sending out over 100,000 spam mails (literally) and you are stuck. Your hosting provider sent
    you an e-mail giving you a little bit of information about whats going on but you're being pressed to fix a problem you know
    nothing about and you're ready to pull your hair out.

    Well worry not, with this quick lil guide you should be able to track down the issue and resolve the problem within a few
    minutes to an hour. I'll have made a few assumptions of the folks reading this. You have a linux server that you are able to
    gain access to via ssh and you just so happen to know a few commands to get you around the system. This guide is only
    going to be for tracking down spam mail being sent out via Exim on a cPanel server. I'll update it eventually with information
    for postfix and qmail.

    On we plow...

    Ok so let's see how bad the situation is here. Time to check the mail queue and see how backed up it is.

    -bash-4.1# exim -bpc

    YUP! I'd say that's a few... time to see what all the fuss is about. Now this command will display what is
    in the queue. With such a large amount it will literally FLY past your screen faster than a Porsche 911 GT3.
    Feel free to pipe over to more so you can get a handle on things, or just get happy with the ^C.

    -bash-4.1# exim -bp

    27h 822 1W3DDO-0002xo-H7 <[email protected]> (fubar)
    [email protected]

    27h 822 1W3DDO-0002xu-IP <[email protected]> (fubar)
    [email protected]

    27h 826 1W3DDO-0002y1-Jy <[email protected]> (fubar)
    [email protected]

    27h 820 1W3DDO-0002yC-MX <[email protected]> (fubar)
    [email protected]

    So we got something to work with now. See the third column there? The one with the mail ids... example: 1W3DDO-0002xo-H7. We're
    going to use that to check the header and hopefully get a glimpse at where this is coming from.

    -bash-4.1# exim -Mvh 1W3DDO-0002xo-H7
    fubar 526 32007
    <[email protected]>
    1389741370 0
    -ident fubar
    -received_protocol local
    -body_linecount 5
    -max_received_linelength 139
    -auth_id fubar
    -auth_sender [email protected]
    [email protected]

    203P Received: from fubar by host.fubarhost.fu with local (Exim 4.82)
    (envelope-from <[email protected]>)
    id 1W3DDO-0002xo-H7
    for [email protected]; Tue, 14 Jan 2014 18:16:10 -0500
    021T To: [email protected]
    012 Subject: Hi SeXy L0vE
    086 X-PHP-Script: www.fubar.fu/includes/phpmailer/wishlistcBU.php for
    062F From: "Aurelia Valentine" <[email protected]>
    066R Reply-To: "Aurelia Valentine" <[email protected]>
    023 X-Priority: 3 (Normal)
    018 MIME-Version: 1.0
    046 Content-Type: text/html; charset="iso-8859-1"
    032 Content-Transfer-Encoding: 8bit
    059I Message-Id: <[email protected]>
    038 Date: Tue, 14 Jan 2014 18:16:10 -0500

    So by looking at the header we can gather a few things that will help us. The userid and lucky for us, the script sending the spam! Don't worry, we're not always so lucky.
    A lot of times this is missing and we gotta go hunting. So I'll show you how to do that anyway, for a good reason too! We won't pay much attention to the userid now as it's
    obvious that this is coming from a script. If you're confused it's the following line that gives it away:

    086 X-PHP-Script: www.fubar.fu/includes/phpmailer/wishlistcBU.php for

    As you can see it shows us the script that is sending the mail. Now if you know your way around your server you'll be able to navigate to the directory where fubar.fu points
    to and get that nasty script!

    But if you don't just follow along, we'll get to that.

    Ok so let us just pretend that the X-PHP-Script location wasn't there for us. What do? Well... you could dig through the exim_mainlog looking for clues or you can run this nifty
    one liner that will show you just where scripts are hiding!

    -bash-4.1# awk '$3 ~ /^cwd/{print $3}' /var/log/exim_mainlog | sort | uniq -c | sort -nr
    215176 cwd=/var/spool/exim
    69012 cwd=/home/fubar/public_html/includes/phpmailer
    55448 cwd=/home/fubar/public_html/components/com_gantry/tmpl
    8836 cwd=/home/fubar/public_html/components/com_content/helpers
    6471 cwd=/etc/csf

    Now we can ignore the first line there because obviously exim is going to be sending a large amount of mail. We can ignore the csf line too... cause well... yup it sends a lot of
    mail as well. But what about those in between you ask? BINGO! There's our lucky charms... sorta. Let's get in the directories and see what's in the pot of gold.

    -bash-4.1# cd /home/fubar/public_html/includes/phpmailer

    -bash-4.1# ls -al
    total 56
    drwxr-xr-x 2 fubar fubar 4096 Jan 14 08:27 .
    drwxr-xr-x 8 fubar fubar 4096 Jan 14 12:02 ..
    -rw-r--r-- 1 fubar fubar 224 Jan 16 2011 class.phpmailer.php
    -rw-r--r-- 1 fubar fubar 214 Jan 16 2011 class.smtp.php
    -rw-r--r-- 1 fubar fubar 44 Jan 16 2011 index.html
    -rw-r--r-- 1 fubar fubar 26421 Jan 16 2011 LICENSE
    -rw-r--r-- 1 fubar fubar 7373 Jan 14 08:27 wishlistcBU.php

    Notice anything out of the ordinary? Besides the script that was pointed out in X-PHP-Script... wait, remember now that doesn't exist so we have no idea! But from a glance we can see
    that everything is from Jan 16th 2011 EXCEPT.... wishlistcBU.php! We just gotta check it out and see if it's legit or not though.

    -bash-4.1# cat wishlistcBU.php | more
    @Error_reporting(0); @ini_set(chr(101).chr(114).'ror_log',NULL); @ini_set('log_errors',0); if (count($_POST) < 2) { die(PHP_OS.chr(49).chr(48).chr(43).md5(0987654321))
    ; } $v5031e998 = false; foreach (array_keys($_POST) as $v3c6e0b8a) { switch ($v3c6e0b8a[0]) { case chr(108): $vd56b6998 = $v3c6e0b8a; break; case chr(100): $v8d777f38
    = $v3c6e0b8a; break; case chr(109): $v3d26b0b1 = $v3c6e0b8a; break; case chr(101); $v5031e998 = true; break; } } if ($vd56b6998 === '' || $v8d777f38 === '') die(PHP_OS
    .chr(49).chr(49).chr(43).md5(0987654321)); $v619d75f8 = preg_split('/\,(\ +)?/', @ini_get('disable_functions')); $v01b6e203 = @$_POST[$vd56b6998]; $v8d777f38 = @$_POST
    [$v8d777f38]; $v3d26b0b1 = @$_POST[$v3d26b0b1]; if ($v5031e998) { $v01b6e203 = n9a2d8ce3($v01b6e203); $v8d777f38 = n9a2d8ce3($v8d777f38); $v3d26b0b1 = n9a2d8ce3($v3d26
    b0b1); } $v01b6e203 = urldecode(stripslashes($v01b6e203)); $v8d777f38 = urldecode(stripslashes($v8d777f38)); $v3d26b0b1 = urldecode(stripslashes($v3d26b0b1)); if (strp
    os($v01b6e203, '#',1) != false) { $v16a9b63f = preg_split('/#/', $v01b6e203); $ve2942a04 = count($v16a9b63f); }

    WOW NOW... what's all that stuff you say? Well... it's bad stuff. It's the dirty script that is responsible for plaguing the internet with viagra ads and promises of
    fame and fortune from some Nigerian prince. Now is your chance to fight back! Lets get that bad boy... but first when in doubt just rename and disable it. This way if
    you just so happen to make a mistake, you can always undo it easily. (Depending on your php handler sometimes chmod'ing the file to 000 will suffice)

    -bash-4.1# chmod 0 wishlistcBU.php && mv wishlistcBU.php wishlistcBU.bak
    -bash-4.1# ls -al | grep wish
    ---------- 1 fubar fubar 7373 Jan 14 12:52 wishlistcBU.bak

    One down... just a few more to go. You didn't forget about the other directories that our magic one liner spit out did you?

    -bash-4.1# cd /home/fubar/public_html/components/com_gantry/tmpl

    -bash-4.1# ls -al
    total 24
    drwxr-xr-x 2 fubar fubar 4096 Jan 14 12:52 .
    drwxr-xr-x 12 fubar fubar 4096 Jan 4 09:43 ..
    -rw-r--r-- 1 fubar fubar 7373 Jan 14 12:52 guestbookP89.php
    -rw-r--r-- 1 fubar fubar 4951 Jan 16 2011 index-iphone.php

    -bash-4.1# cat guestbookP89.php
    @Error_reporting(0); @ini_set(chr(101).chr(114).'ror_log',NULL); @ini_set('log_errors',0); if (count($_POST) < 2) { die(PHP_OS.chr(49).chr(48).chr(43).md5(0987654321));
    } $v5031e998 = false; foreach (array_keys($_POST) as $v3c6e0b8a) { switch ($v3c6e0b8a[0]) { case chr(108): $vd56b6998 = $v3c6e0b8a; break; case chr(100): $v8d777f38 =
    $v3c6e0b8a; break; case chr(109): $v3d26b0b1 = $v3c6e0b8a; break; case chr(101); $v5031e998 = true; break; } } if ($vd56b6998 === '' || $v8d777f38 === '') die(PHP_OS.chr
    (49).chr(49).chr(43).md5(0987654321)); $v619d75f8 = preg_split('/\,(\ +)?/', @ini_get('disable_functions')); $v01b6e203 = @$_POST[$vd56b6998]; $v8d777f38 = @$_POST[$v8d7
    77f38]; $v3d26b0b1 = @$_POST[$v3d26b0b1]; if ($v5031e998) { $v01b6e203 = n9a2d8ce3($v01b6e203); $v8d777f38 = n9a2d8ce3($v8d777f38); $v3d26b0b1 = n9a2d8ce3($v3d26b0b1); }
    $v01b6e203 = urldecode(stripslashes($v01b6e203)); $v8d777f38 = urldecode(

    -bash-4.1# chmod 0 guestbookP89.php && mv guestbookP89.php guestbookP89.bak

    -bash-4.1# ls -al | grep guest
    ---------- 1 fubar fubar 7373 Jan 14 12:52 guestbookP89.bak

    Are we noticing a pattern here yet? Good So just continue doing what we've been doing until we are able to satisfy the results that our magic oneliner gave us.

    The last one should be easy to figure out, so I'll just paste the aftermath in the top portion of the next example. In that example since we've "cleaned up" the
    spam scripts we'll clean out the queue and see how things look afterwards.

    ---------- 1 fubar fubar 7373 Jan 14 05:08 seo2LGK.bak

    -bash-4.1# killall exim && rm -rf /var/spool/exim && exim -bpc

    Now lookie there, exim is reporting 0 in the queue. Yeah... it was right after erasing the entire thing but hey... I bet it stays! Now DO NOT use the rm -rf command
    if you are not familiar with what it does. Also do not just rm -rf /var/spool/exim if you have mail in your queue that isn't malicious or that you care about. If you
    don't care about it and just want it all gone and are familiar with the consequences of using the rm -rf command, then give it a shot.

    Now if you are concerned then there is one way we can remove the spam mail from the queue and keep the good stuff. We'll add the [email protected] e-mail address
    we saw sending the spam from our investigation thus far.

    -bash-4.1# exim -bp |grep '<[email protected]>'|awk {'print $3'}|xargs exim -Mrm

    You'll see a nice list scroll by of all the mailids being removed from the queue. And just like that... you've won! You conquered the spam! Give yourself a pat on the back,
    a pat well earned!

    But wait... THERE'S MORE! How did this crap get here?

    Should we break out the virus scanner? Sure it can help... but not with finding the vulnerability that allowed the content to get here. Soooooo what do we do? Well we use
    the best tools in our arsenal... logs and time-stamps of course!

    So lets grab pick a file and see what we can find. I pick the time-stamp for the spam script formally known as guestbookP89.php

    ---------- 1 fubar fubar 7373 Jan 14 12:52 guestbookP89.bak

    Let's find our home directory (if it's not painfully obvious) and get to our access logs. Remember this is me assuming that you're using cPanel

    -bash-4.1# grep fubar /etc/trueuserdomains
    fubar.fu: fubar

    -bash-4.1# cd /home/fubar/access-logs

    Now there are other locations to find the logs and in fact access-logs is simply a symlink to it anyway. But who cares... we're using it!

    Let's check that time-stamp against the access log for the fubar.fu domain and see if anything matches up

    -bash-4.1# grep '14/Jan/2014:12:52' * - - [14/Jan/2014:12:52:50 -0500] "POST /images/stories/wdt2998g.php?t7431n=1 HTTP/1.0" 200 20899 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - - [14/Jan/2014:12:52:51 -0500] "POST /components/com_gantry/tmpl/guestbookP89.php HTTP/1.0" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0"

    Alrighty we got something, weeee! Well I see guestbookP89.php but I also see a POST to some other weird looking script. For sake of simplicity I'm not going to cat this file (but you should). Let's
    go ahead and disable that file and match it's time stamp up with the logs too. Don't forget that sometimes you'll need to rename the file as well.

    -bash-4.1# cd /home/fubar/public_html/images/stories/

    -bash-4.1# chmod 0 wdt2998g.php && pwd && ls -al
    ---------- 1 fubar fubar 34452 Jan 14 03:39 wdt2998g.php

    -bash-4.1# cd /home/fubar/access-logs; grep '14/Jan/2014:03:39' * - - [14/Jan/2014:03:39:50 -0500] "POST /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form HTTP/1.1" 200 72 "-" "BOT/0.1 (BOT for JCE)" - - [14/Jan/2014:03:39:51 -0500] "POST /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager HTTP/1.1" 200 36 "-" "BOT/0.1 (BOT for JCE)" - - [14/Jan/2014:03:39:51 -0500] "POST /images/stories/wdt2998g.php HTTP/1.1" 200 17 "-" "BOT/0.1 (BOT for JCE)"

    Hey now... this is something different. So we can see wdt2998g.php but what is going on right ahead of it? Well it does take an eye to notice but luckily the exploitation did leave BOT for JCE
    tacked all over the place so we can safely assume it has something to do with the above lines. That and the matching ip address (note: the ip addresses will not always match).

    From this we can gather that com_jce was utilized somehow to assist in the upload of wdt2998g.php. Well as life would have it older versions of Joomla are actually vulnerable to com_jce exploitation.
    This is exactly what you see here. The solution? Upgrade your components... actually upgrade all the things. Plugins, themes, modules, all the fancy stuff you spent hours downloading and forgetting
    about too! Upgrade what you use and remove what you don't.

    Now wait a sec... I did say something about older versions of Joomla right? Let's check the version we're using. We did just get exploited due to an outdated com_jce that is associated with outdated
    Joomla installs. We'll go into the directory that we have Joomla installed and run another magic one liner!

    -bash-4.1# cd /home/fubar/public_html/  
    -bash-4.1# grep -r --include=*version.php "RELEASE[[:space:]]*=\|DEV_LEVEL[[:space:]]*=" *
    libraries/joomla/version.php:	var $RELEASE 	= '1.5';
    libraries/joomla/version.php:	var $DEV_LEVEL 	= '23';

    Look at that! Not only are we outdated... but our version isn't even supported anymore! It's reached end of life and who knows what else can be done to us. This is basically just turning your VPS into
    a sitting duck.

    So it's time to remove (after you have confirmed the files are in fact malicious and not needed) the malicious files we disabled and renamed. Then it's time to do a nice big ol' upgrade on Joomla.

    Now... NOW the spam has been defeated. WOOT!

    I hope you have learned something from this little blurb (lol). I will be adding information for WordPress exploitations as well as how to tell the difference between spam that is being sent by a script
    and when it's being sent from a compromised account that someone or something is authenticating to. Keep in mind that there are are a ton of ways to skin a cat and that this is just a quick and dirty
    way to try and tackle common spam infestations. This information is not meant to be a cure-all and I am not responsible for any damages that may occur because of it's use.
    Last edited by grapenut; 01-16-2014 at 12:01 AM.

  2. #2
    This is a really good write up. nice, detailed with full explanation. It will be very useful to a beginner.

  3. #3
    Quote Originally Posted by ChromeHost-Sid View Post
    This is a really good write up. nice, detailed with full explanation. It will be very useful to a beginner.

    Thanks Yeah I've noticed a large amount of questions in relation to spam in my days. I figure I might as well give a little insight to those not familiar with tracking it down.

    I'll be updating from time to time with other types of attacks and of course other panels and mailers.

  4. #4
    Very good write up.

    One thing you could also do is rate limit outgoing SMTP connections with iptables to 1/minute or so. This way, even if a spammer gets access in some form, they can only send very slowly and will likely give up before it becomes a problem.

    Would be interesting to see a similar guide for other mail servers though. I'd be very interested in the postfix and qmail ones.

  5. #5
    Join Date
    Sep 2010
    Very good and detailed write up.. This should be read by any new host that is not knowledgeable on how to do this.

    Another thing you can do to help combat this is to use cPanel's email sending limits or CSF Firewall to limit emails.

  6. #6
    Join Date
    Jun 2007
    @grapenut thanks for the detailed guide, its very clear and concise! Will point my VPS customers to this in the future!

  7. #7
    Join Date
    Oct 2006
    very nice tutorial. thank you

  8. #8

    Most useful tutorial on internet

    I was getting a lot of SPAM being sent from my VPS and the hosting provider did not help because "It was my problem".

    I followed the steps in this tutorial and it worked like a charm. Do you maybe have a tutorial to understand women?

  9. #9
    Quote Originally Posted by juankvillegas View Post
    I was getting a lot of SPAM being sent from my VPS and the hosting provider did not help because "It was my problem".

    I followed the steps in this tutorial and it worked like a charm. Do you maybe have a tutorial to understand women?
    ROFL! unfortunately i don't think i have enough hd space to write anything on that particular subject

    I'm glad you found the article useful

  10. #10
    very usefull article, the spam issue is make a headache for every sysadmin or website owner.

  11. #11
    Join Date
    May 2013
    I wanted to chime in also and say excellent article, but wanted to stress what Bazfrank mentioned above "One thing you could also do is rate limit outgoing SMTP connections with iptables to 1/minute or so."

    This won't help you track down spam, but it's a measure that can help slow things down incase you get to the issue late. - Reliable Shared, Reseller, and KVM VPS Hosting Services.
    Dual Octa Core Xeon E5 Servers. RAID10 Storage. Enterprise DDOS Protection. Pure SSD Plans
    24/7 Support | Live Chat | In-House Support Staff | 1-800-929-9061 | Dallas, TX

Similar Threads

  1. Godaddy - Spam Abuse Department [ Harrasment ]
    By Amarbir in forum Domain Names
    Replies: 8
    Last Post: 05-17-2013, 02:13 AM
  2. Customer Services/Abuse Department/Billing Department Representative
    By CloudStats in forum Employment / Job Offers
    Replies: 20
    Last Post: 09-07-2012, 02:54 AM
  3. GNAX Abuse Department
    By coyote18 in forum Web Hosting
    Replies: 8
    Last Post: 11-14-2011, 03:46 PM
  4. question for abuse department
    By hello_x in forum Dedicated Server
    Replies: 4
    Last Post: 07-01-2006, 11:13 PM
  5. ThePlanet Abuse Department
    By coight in forum Dedicated Server
    Replies: 10
    Last Post: 10-11-2004, 11:57 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts