Results 1 to 10 of 10
  1. #1
    Join Date
    Aug 2002
    Location
    Kelowna B.C.
    Posts
    1,687

    An interesting quandry....

    This afternoon we caught a spammer using our system as a relay for hundreds of other machines through a php script that was uploaded through a script exploit on one of our clients websites.

    Now the problem....

    We've deleted the website, the DNS records on our server, and the doman registrar however despite these changes the hundreds of machines out there which used our server for thr relaying still find their way on to our server. This causes quite a large load issue as 50 requests per second are generated on this now non existant script.

    The server load is not high, but our server shows 500 currently running processes, which tends to eat away at memory. Typically this server runs a steady 250 processes. I find i'm having to run several killall's on httpd throughout the day to free up the memory again once it dives into the swap.

    23:51:53 up 6:20, 2 users, load average: 0.73, 2.84, 4.01
    561 processes: 560 sleeping, 1 running, 0 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 2.6% 0.0% 1.8% 0.0% 0.0% 1.5% 93.8%
    cpu00 1.7% 0.0% 1.7% 0.0% 0.0% 1.3% 95.0%
    cpu01 3.5% 0.0% 1.9% 0.0% 0.0% 1.7% 92.6%
    Mem: 2055452k av, 1990016k used, 65436k free, 0k shrd, 114048k buff
    1357124k actv, 263572k in_d, 33200k in_c
    Swap: 1044144k av, 390708k used, 653436k free 372924k cached

    My question is - since all these requests are coming from completely different IP addresses, there is no possible way to block them all without seriously affecting our other clients. Is there a way to deny acccess to a particular file on a server without having to generate a log for the denial (and therefore create an apache process) - but instead it just ignores all traffic to that URL?

    I'm almost certain this was more an attack with the spam as a bonus. I don't believe this user actually has control over all those IP addresses being used from around the world - it's almost like this is an evil fakezilla version used to generate non-existant traffic.....

    [Wed May 4 00:02:51 2005] [error] [client 24.152.245.243] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 24.55.244.239] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 24.55.244.239] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 207.59.45.83] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 207.59.45.83] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 80.250.181.17] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 80.250.181.17] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 220.225.96.54] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 220.225.96.54] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 195.209.66.199] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 195.209.66.199] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 194.85.140.36] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 194.85.140.36] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 213.134.211.170] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 213.134.211.170] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 82.54.112.91] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 82.54.112.91] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 81.0.88.96] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 81.0.88.96] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 81.73.189.250] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 81.73.189.250] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 213.186.187.162] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 213.186.187.162] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 193.219.58.58] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 193.219.58.58] File does not exist: /var/www/html/404.shtml
    [Wed May 4 00:02:51 2005] [error] [client 217.132.79.210] File does not exist: /var/www/html/webadm/web.php
    [Wed May 4 00:02:51 2005] [error] [client 217.132.79.210] File does not exist: /var/www/html/404.shtml


    Your Thoughts would be appreciated....thanks
    Hosting.Express | Affordable Web and Email Hosting
    Shared | Reseller | 24/7 Support | NSA Free
    SPECIAL OFFER - domain name, email and cPanel web hosting = $3.73 per month | Contact Us: 1-800-861-1888

  2. #2
    Join Date
    Oct 2003
    Location
    Long Island, New York
    Posts
    220
    Write a little script to grep through the logs and add blocks for those IPs using iptables. It looks like a little http get DDOS to me...
    TWSites.com - Business Web Hosting Solutions & Server Management Since 2003

  3. #3
    Join Date
    Feb 2003
    Location
    CT
    Posts
    481
    Or use mod_security to block requests to web.php.

  4. #4
    Hi Hostkookster,

    We had the same problem on one of our servers.

    What did you do you completely resolve this issue?
    Was your server compromised?
    What exactly was going on?

    Appreciate any info.

    Regards,
    Suhail.
    OSHS Ltd
    OSHS Services - DNS Clusters | R1Soft Licenses | Remote Backup Storage | R1Soft CDP Storage | Cheap Dedicated Servers
    EconDC.com - Enterprise UK Server Colo & Rack Space at Lowest Prices

  5. #5
    Greetings:

    Check if your server is an open relay.

    As root on the server, run telnet relay-test.mail-abuse.org

    And then go through any failures to determine if a failure is a false positive.

    Make sure your box is secured and kept secured (server hardening is not once and done but a daily activity).

    As part of the server security, make sure you are using multiple layers so that if a given layer is weaker (like client scripts), you still have some protection.

    These include, but are not limited to, securing /tmp (linking /var/tmp to /tmp) and /dev/shm (if Linux-based system), removing group and world access from all fetch like programs and compilers, using mod_security from http://www.modsecurity.org/ along with a good set of rules geared towards your automation systems and client base needs.

    Thank you.
    ---
    Peter M. Abraham
    LinkedIn Profile

  6. #6
    Hi,

    This problem STILL exists on our servers.

    Firstly our server isn't an open relay - this has been thoroughly tested.

    Secondly, using mod_security we cannot block the HTTP requests.

    The only solution has been that we removed the DNS for the site affected, and this stopped the HTTP requests. Of course this has made the site unaccessible.

    Hostkookster - do you have an update on what steps you have taken and whether you've been successful or not?
    OSHS Ltd
    OSHS Services - DNS Clusters | R1Soft Licenses | Remote Backup Storage | R1Soft CDP Storage | Cheap Dedicated Servers
    EconDC.com - Enterprise UK Server Colo & Rack Space at Lowest Prices

  7. #7
    If you are using linux, you can block connections with iptables and string match of URL accessed or part of it.

  8. #8
    Yeah, the problem is that we just sorted the number of IPs involved, and we're talking about over 30,000 unique IPs that has tried accessing this one URL. So how exactly do you configure iptables to handle this?
    OSHS Ltd
    OSHS Services - DNS Clusters | R1Soft Licenses | Remote Backup Storage | R1Soft CDP Storage | Cheap Dedicated Servers
    EconDC.com - Enterprise UK Server Colo & Rack Space at Lowest Prices

  9. #9
    Originally posted by suhailc
    Yeah, the problem is that we just sorted the number of IPs involved, and we're talking about over 30,000 unique IPs that has tried accessing this one URL. So how exactly do you configure iptables to handle this?
    Simply,
    if ("iptables+string match" == "URL")
    block;
    else
    letIn;

  10. #10
    Join Date
    Sep 2000
    Location
    Alberta, Canada
    Posts
    3,109
    Simple is best, whenever possible.

    File does not exist: /var/www/html/404.shtml
    File does not exist: /var/www/html/webadm/web.php

    Presuming 'webadm' is a Client account which you want to keep running, then simply create a "404.shtml" file, which should always be done BTW for every account on the Server, and in the .htaccess file for the account you setup a Redirect.

    Redirect Gone /web.php


    Although this will not stop requests for the file in question, it will Allow Apache to (very quickly) process the request then drop the connection. It will also prevent; huge error log files, resources wasted on creating those useless error log entries, and as a Bonus, prevent Search Engines from providing the URL within their databases.

    Just a thought.
    PotentProducts.com - for all your Hosting needs
    Helping people Host, Create and Maintain their Web Site
    ServerAdmin Services also available

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •