Results 1 to 22 of 22
  1. #1
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190

    Apache Expert Needed

    I am working with some clients who've had their web server hacked. It appears that maybe an Apache process is running that's only being active X out of 100 requests.

    It's been very difficult to troubleshoot so I'm looking for some answers.

    First of all, when people access a website on the infected webserver, it only serves up the malicious javascript once in awhile. We can't make it show it's ugly head with any consistency.

    When it does, it shows the server in the headers as: Apache. When the infection isn't visible the headers show: Apache/2.2.0 (Fedora). The date in reply headers is correct when the infection isn't active. When the infection is active, it shows the wrong date and it's always: Date: Sun, 27 Sep 2009 08:49:10 GMT.

    It almost appears that there's an infectious Apache (httpd) child that's running. Is it possible or does Apache always run an exact copy of itself? I'm grasping at straws here, but can anyone shed light on this?

    Unfortunately I don't have shell access (unlike the hackers) to any infected webserver but I will try anything to find out what it is.

    I've been reading Denis' blog on Unmaskparasites.com and nothing with this one seems to be the same.

    Anyone?

    Thank you in advance.

  2. #2
    Join Date
    Oct 2006
    Location
    /usr/src/linux/
    Posts
    699
    It may be an apache module, issue httpd -l see if there are some phishy modules loaded. Also check root's cron, issue crontab -u root -l see if something is killing httpd and starts its own malicious httpd server. In both scenarios the attacker would need root access, if I were you I'd backup all data and rebuild the server then analyze each site one by one.
    VPSnoc.com offers high quality Xen OpenVZ & Windows Virtual Private Servers at affordable prices.
    99.95% Uptime | 24/7/365 Support | Unmetered bandwidth.
    Follow us: twitter.com/VPSnoc

  3. #3
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    Quote Originally Posted by WeWatch View Post
    Unfortunately I don't have shell access (unlike the hackers) to any infected webserver but I will try anything to find out what it is.
    You have to find the PHP backdoor (look inside your FTP log), the for the rest you need the admin to solve the problem.
    I've been dealing 3 times with that virus for the last few days with my customers.
    The backdoor is used to inject a script + ELF binary. The script does some tests, check Apache user:group and then will run the binary with apache privileges. Then both the script and ELF are deleted from disk.
    That's a beladen-like exploit except that now it doesn't inject any obfuscated code inside PHP files, instead the binary uses a bug in Apache APR.
    Check your cookies as well, one of them is used to trigger the redirection.
    Restarting Apache will kill the process, but the hacker will come back sooner or later to inject a new one.

    Only root can solve the problem, so if you don't have any access to the server, get in touch with the admin.

    Disable PHP exec() would help a lot if you don't need it.
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  4. #4
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190
    Do you have a copy of the .php file used as the backdoor?

    It's probably something with base64_decode in it. Although we've seen files where they (the hackers) have padded the term base64 with other characters like: #####b#######a###s####################e###6##########4... and then the script does a replace on the character to remove it so the script runs.

    Unfortunately, this hosting provider keeps denying they have a problem. Isn't that the first step toward recovery? Is admitting you have a problem? (a little humor)

  5. #5
    Join Date
    Sep 2007
    Posts
    368

    *

    Quote Originally Posted by WeWatch View Post
    Do you have a copy of the .php file used as the backdoor?

    It's probably something with base64_decode in it. Although we've seen files where they (the hackers) have padded the term base64 with other characters like: #####b#######a###s####################e###6##########4... and then the script does a replace on the character to remove it so the script runs.
    Hi,

    Can you please tell us in which file above change you found?

  6. #6
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190
    It was a randomly named file: 857233.php in the images folder.

    The .php file stared with:

    (usual .php opening) $PyIqJDl='#####e###################v###a######l(b...

    Then the rest as I posted previously. It was a binary (ELF) that it obfuscated in the base64 code. The .php file ended with:

    $PyIqJDl=str_replace('#', '', $PyIqJDl);$OOxbqtu=create_function('',$PyIqJDl);$OOxbqtu(); ?>

  7. #7
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    This is extremely common. Did you check the ftp logs?

    It almost appears that there's an infectious Apache (httpd) child that's running. Is it possible or does Apache always run an exact copy of itself? I'm grasping at straws here, but can anyone shed light on this?
    go to google and search:
    site:webhostingtalk.com flame.so

    could be a similar problem.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  8. #8
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    They are using some known PHP conf files (mostly from CMS), then there's a single line on the top which will check the POST parameters.

    p = pass
    c = base64 encoded script + ELF

    Thus, there's a "eval(base64_decode"

    Those files were uploaded months ago via FTP (march/april) and in several cases, left on the server for a later use. In one case out of 3, right before dying the script chmoded the backdoor to 0000. That means that it could only be read from shell.
    After the process is up and running, the backdoor is not deleted, the hacker expect to use it again later.
    I setup honeypots on the 3 servers and wait for him to come back.


    Unfortunately, this hosting provider keeps denying they have a problem. Isn't that the first step toward recovery? Is admitting you have a problem? (a little humor)
    One hosting company said they checked everything and that the virus was on the visitors computer. I found 3 shell scripts, the backdoor + running process, 2 trojans and even a win32 virus sitting in the /tmp folder !
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  9. #9
    Join Date
    Feb 2006
    Location
    Buffalo NY
    Posts
    1,348
    Try to check most recently modified files..

    Code:
    find /path/to/web/stuff -name '*.php' -mtime -1
    Cody R.
    Hawk Host Inc. Proudly Serving websites since 2004.
    Let's Encrypt Sponsor.

  10. #10
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190
    The problem is that I don't have access to all the websites on that shared server. I only have access to one. I have checked for recently modified files, but nothing shows. This infection seems to affect all the websites on a shared server.

  11. #11
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    Got him ! He fell into my honeypot.

    Code:
    201.233.216.14 - - [30/Sep/2009:18:24:51 +0100] "POST /test/php/test.php HTTP/1.1" 200 281 "http://www.google.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"
    72.12.166.145 - - [30/Sep/2009:18:24:52 +0100] "POST /test/php/test.php HTTP/1.1" 200 8976 "http://www.google.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"


    So you are, as well as everyone reading this, higly advised to monitor any POST request with :

    - http://www.google.com as HTTP_REFFER
    - Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar) as USER_AGENT

    Both have been used for injection of the code since March.
    Note that there are always 2 consecutive POST requests within 1 or 2 seconds, and the bot comes back on a daily basis.
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  12. #12
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190
    Do you have any packet captures by chance?

    Is it always Google as the referer?

    Did you check any of the websites on that server for serving up the infectious code after you saw these POSTs? In other words, can we say for certain that these POSTs triggered the process of intercepting the legit HTTP requests and returning the infectious scripts in the HTTP response?

    Awesome job!

    Thank you for sharing.

    Do you have the files that these POSTs were being sent to?

    Please share. Inquiring minds want to know...

  13. #13
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190
    The first IP address is from Medellin, Columbia, the second one from Canada.

    Just an FYI...

  14. #14
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    Yes, referer + browser are always identical.
    The 2 POST requests only submit the base64 encoded script, ELF, commands.
    The ELF will trigger the redirection (it will first set a 'n_sess_id' cookie and then will redirect according to the expiration date, it think, as soon as the user come back or click on another link).

    As it comes everyday just check your access logs, you can't miss it. It you don't find it, the backdoor is inside another vhost (it affects the whole server).

    I haven't got the files yet as I need to SSH to the server later today or tomorrow to retrieve them.

    The PHP backdoor was uploaded by FTP.
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  15. #15
    Join Date
    Feb 2006
    Location
    Buffalo NY
    Posts
    1,348
    Quote Originally Posted by khunj View Post
    Yes, referer + browser are always identical.
    The 2 POST requests only submit the base64 encoded script, ELF, commands.
    The ELF will trigger the redirection (it will first set a 'n_sess_id' cookie and then will redirect according to the expiration date, it think, as soon as the user come back or click on another link).

    As it comes everyday just check your access logs, you can't miss it. It you don't find it, the backdoor is inside another vhost (it affects the whole server).

    I haven't got the files yet as I need to SSH to the server later today or tomorrow to retrieve them.

    The PHP backdoor was uploaded by FTP.
    Sounds pretty much identical to the issues we have very often - a few virus' (gumblar, etc) log FTP logins and upload these type of scripts.. it's mildly obnoxious.

    I'm tempted to say if you used something like Snort with the proper rules it would have found this fairly easily / quickly, though it's difficult to know. If you want we can do some testing just for the sake of knowing .
    Cody R.
    Hawk Host Inc. Proudly Serving websites since 2004.
    Let's Encrypt Sponsor.

  16. #16
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    I think a simple mod_security rule should do the trick (POST request with a google.com referer + user_agent =~ /alexia/)

    The backdoor is pretty old, but the injected code is quite tricky.
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  17. #17
    Join Date
    Feb 2006
    Location
    Buffalo NY
    Posts
    1,348
    Quote Originally Posted by khunj View Post
    I think a simple mod_security rule should do the trick (POST request with a google.com referer + user_agent =~ /alexia/)

    The backdoor is pretty old, but the injected code is quite tricky.
    That should block anyone who was referred from Google and who had the Alexa toolbar installed - very bad idea.

    I was referring to using Snort to find the issue / script as there was difficulty in doing so
    Cody R.
    Hawk Host Inc. Proudly Serving websites since 2004.
    Let's Encrypt Sponsor.

  18. #18
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    I don't know Alexa toolbar, but that would be quite odd if it sent 2 consecutive POST requests to a PHP script from google ?

    Looking inside access_log, or searching for "eval(base64_decode" string in PHP files works pretty well.
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  19. #19
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190
    From what we've gathered, the consecutive POSTs are typically from different IP addressses but to the same .php file. That may cause issues with Snort. Also, keep in mind that sometimes the eval(base64_decode is obfuscated with characters padding the string as posted previously.

    I'm not a Snort expert, I'm just sayin'.

  20. #20
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    mod_security rule (didn't test it but should be OK :


    Code:
    SecRule REQUEST_METHOD "^POST$" "chain,log,drop"
    SecRule REQUEST_HEADERS:User-Agent "Alexa Toolbar" "chain"
    SecRule HTTP_REFERER "www\.google\.com" "chain"
    SecRule REQUEST_URI "\.php" "t:none"
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  21. #21
    Join Date
    Oct 2008
    Location
    Chicago, IL
    Posts
    190
    But wouldn't that block on anyone POSTing to a php form from a Google search with Alexa toolbar installed? That sounds like a candidate for false positives.

    Then all the hackers would have to do is remove the Alexa toolbar from their user-agent and they'll slide right by.

    I don't know what the answer is, I'm just sharing my thoughts.

  22. #22
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    It will drop them. Replace 'drop' with 'allow' to let them go and watch your mod_security log as their POST requests will be logged.

    But in any case, by looking through FTP and HTTP (access/error) logs it is really easy to find out where is the backdoor.
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

Similar Threads

  1. MYSQL redhat apache expert needed.
    By bacanak in forum Systems Management Requests
    Replies: 4
    Last Post: 10-09-2008, 11:11 AM
  2. Linux/Apache/Server Expert Needed for Utilization
    By INTEL in forum Employment / Job Offers
    Replies: 2
    Last Post: 05-01-2005, 08:41 PM
  3. mysql/php scripter/linux/apache expert needed
    By streamservice.org in forum Employment / Job Offers
    Replies: 4
    Last Post: 03-05-2005, 10:52 AM
  4. Expert Needed. Apache. PHP.
    By INTEL in forum Employment / Job Offers
    Replies: 6
    Last Post: 08-18-2004, 07:06 AM
  5. mod_layout/Apache expert needed
    By Tox in forum Employment / Job Offers
    Replies: 1
    Last Post: 09-19-2003, 04:20 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •