Results 1 to 22 of 22
  1. #1
    Join Date
    Jul 2001
    Posts
    892

    Am I being DoSed? How do i Make it STOP!?!

    One problem after another, and my admin is offline.

    I go to update my site, and FTP times out. I go to visit my site and the page loads, but very slowly.

    I log into SSH (slow, again) and check top

    10:28pm up 1 day, 3:03, 1 user, load average: 119.84, 106.30, 85.30
    437 processes: 417 sleeping, 20 running, 0 zombie, 0 stopped
    CPU states: 76.8% user, 23.1% system, 0.0% nice, 0.0% idle
    Mem: 506236K av, 500804K used, 5432K free, 0K shrd, 2760K buff
    Swap: 1048280K av, 596680K used, 451600K free 43440K cached
    not good. apache is the only user and httpd the only command that I see.

    netstat -ta

    pretty well every connection is from a different source, and slightly more are TIME_WAIT than ESTABLISHED

    My server is grinding to a halt, and I dont know what to do. Stuff like this always seems to happen during the rare instance that my admin is offline. It's either a conspiracy against me, or Murphy's law.

    Anyways, any insight or help would be greatly appreciated

    =THAnks

  2. #2
    Join Date
    Jun 2000
    Location
    Washington, USA
    Posts
    5,991
    I would stop Apache, for about 10min then start it again. If someone is DoSing you, they may give up.

  3. #3
    Join Date
    Jul 2001
    Posts
    892
    That's an idea, but one would think that after several hours of unsuccessful pounding, the culprits would give up (if it is infact a DOS)

    I'd rather solve this problem with something other than psychology

  4. #4
    Join Date
    Jun 2000
    Location
    Washington, USA
    Posts
    5,991
    Well... are they repeated attempts from the same IPs?

    If not, it could be a site on your server just got popular.

    You could always filter the IPs with IP Tables... that would be a pain though.

  5. #5
    Join Date
    Jun 2000
    Location
    Washington, USA
    Posts
    5,991
    You could also kill some apache processes, it'll help for a little bit.

    What's happened is Apache is sucking up all of your RAM, that doesn't help your load average. The lack of RAM hinders other processes from starting/forking, e.g. SSH or FTP.

  6. #6
    Join Date
    Jul 2001
    Posts
    892
    No, they arent repeated attempts from the same IP.

    It is possible that one of my sites has become popular, but my server should be robust enough to handle it. I was pushing 16 megabit with a similar server, and the load averages were nowhere near as high. Right now im not even doing 1/10 of that.

  7. #7
    Join Date
    Jul 2001
    Location
    Troy, Missouri USA
    Posts
    1,299
    If you are running Portsentry (most cPanel/WHM systems do) stop it and then start it again.

  8. #8
    Join Date
    Jun 2000
    Location
    Washington, USA
    Posts
    5,991
    Hmmm, odd... don't know what to tell you. Something is causing Apache to eat up RAM, aside from people accessing the server. Do any of the sites use PHP or mod_perl?

    Sorry I can't be of much help, I just think a site got popular. Wait, what do your access logs show?

  9. #9
    Join Date
    Jul 2001
    Posts
    892
    no port sentry. I'm running Plesk.

    My site uses php, but not to the extent that it would cause problems.

  10. #10
    Join Date
    Apr 2001
    Location
    Paradise
    Posts
    11,868
    Did you tried with
    RLimitMEM
    RLimitCPU ?
    Shared Web Hosting - Reseller Hosting - Semi-Dedicated Servers - SolusVM/XEN VPS
    LiteSpeed Powered - R1Soft Continuous Data Protection - 24/7 Chat/Email/Helpdesk Support
    Cpanel/WHM - Softaculous - R1soft Backup - Litespeed - Cloudlinux -Site Builder- SSH support - Account Migration
    DowntownHost LLC - In Business since 2001- West/Center/East USA - Netherlands - Singapore

  11. #11
    Join Date
    Nov 2001
    Location
    Canada
    Posts
    1,963
    how about ps -aux ?

  12. #12
    Join Date
    Apr 2002
    Location
    Auckland - New Zealand
    Posts
    1,572
    or even

    ps aux | grep apache

    or

    ps aux | grep user

    etc etc....

  13. #13

    Angry

    same here. Apache 1.3 got overload and crash with 60 TIME_WAIT simultany connections.
    The thing is strange that I tell it in the configuration that after 40 connection it must refuse other one.
    Perhaps Apache 2.0 will correct that ?!

  14. #14
    Join Date
    Jun 2001
    Location
    Germany
    Posts
    69

    Check the apache status page

    Hy,

    enable the server-status handler in your Apache config file, kill -HUP your apache and check the server status page with http://your.domain.com/server-status.

    There you can see which apache process with which request eats up the memory and system power.

    After checking dont forget to disable the server status handler and to restart the apache.

    Greetings
    Oliver

  15. #15
    Join Date
    Jul 2001
    Posts
    892
    excellent. I'll keep all this in mind if it happens in the future. I ended up just stopping apache ofr a minute or two, and restarting it. everything was fine after that.

    thanks for all the help, I really appreciate it.

  16. #16
    Join Date
    Dec 2001
    Location
    QLD, Australia
    Posts
    284
    Hrm,

    A little while back one of the servers I admined got DoSed on port 80 (Apache). Ended up setting up a script that did a netstat -an every minute, found an IP with more than 5 active connections and dynamically added the iptables rule.

    I would post the script but I seem to have since lost it. In either case it worked relatively well and while some clients were blocked accidently we (the Web Host and I) certainly thought it was the smartest move rather than have all 300 domains on the server down.

    Cheers,

    Stuart
    Seekbrain.com - Personal Blog
    E-Currency Australia - Crypto Currency Services

  17. #17

    Lightbulb

    You can try this patch ReadRequestTimeout-directive


    Allows specification of a ReadRequestTimeout, so servers with a large
    Timeout setting can still get rid of clients not sending requests.
    Jimmy

  18. #18
    I suggest you check out user cgi files to make sure they are OK (it's bad that you didn't run ps -aux, but anyway, if it was a php it would be hard to trace)... maybe it was a buggy cgi program/script (infinitive loop etc)... sometimes it happens

  19. #19
    Join Date
    Nov 2001
    Location
    Ann Arbor, MI
    Posts
    2,978
    Originally posted by Perlboy
    A little while back one of the servers I admined got DoSed on port 80 (Apache). Ended up setting up a script that did a netstat -an every minute, found an IP with more than 5 active connections and dynamically added the iptables rule...
    We use a similar script, only it notifies us instead of blocking the IP address. Typically a decent (if there is such a thing) DOS attack will trigger a notification when the server load goes too high, as well.

    Most DOS attacks are going to be SYN floods. Our script doesn't check for active connections, but only requests for connections which would better indicate a SYN flood. There is no point in creating active firewall rules for a SYN flood, as the IP addresses will probably be spoofed.

    If you are going to create active firewall rules, you would need to check for "active connections" like you said, as they would then be using a valid IP address that you could block. However, a lot of browsers will open more than one connection, so I would recommend a much higher number than five. Still, this isn't going to be the majority of your DOS attacks, but they are the easiest to deal with.

    However, woodsheds problems didn't indicate a DOS attack, but only an available/overloaded web server that was receiving requests.
    -Mark Adams
    www.bitserve.com - Secure Michigan web hosting for your business.
    Only host still offering a full money back uptime guarantee and prorated refunds.
    Offering advanced server management and security incident response!

  20. #20
    Join Date
    Dec 2001
    Location
    QLD, Australia
    Posts
    284
    Originally posted by bitserve
    However, woodsheds problems didn't indicate a DOS attack, but only an available/overloaded web server that was receiving requests.
    That's a possibly wrong assumption.

    netstat -ta

    pretty well every connection is from a different source, and slightly more are TIME_WAIT than ESTABLISHED
    This indicates it could possibly be a DDoS attack on the Apache port 80 from spoofed IP addresses or even slaves. The fact the server loads are massive indicates huge spawning of Apache children.

    While it IS possible he is receiving mass traffic, all things considered, the possibility of this causing huge server loads compared relatively to a DDoS is low. How many of us have extremely busy websites (ie. 100's or 1000's of hits a second). I have one website which cops 110,000 hits a day (all of which run through a Perl script) and never see loads above around 0.5 - 1.

    Cheers,

    Stuart
    Seekbrain.com - Personal Blog
    E-Currency Australia - Crypto Currency Services

  21. #21
    Join Date
    Nov 2001
    Location
    Ann Arbor, MI
    Posts
    2,978
    Originally posted by Perlboy
    That's a possibly wrong assumption.
    Possibly wrong, yes. But most likely it was a correct assumption. The indicators were not consistent with a DOS attack.

    This indicates it could possibly be a DDoS attack on the Apache port 80 from spoofed IP addresses or even slaves.
    It does not indicate any such thing. In general, you can not spoof a source IP address and have the packet make it back to you in order to complete a TCP connection and have a TIMED_WAIT. A man in the middle attack would allow this, but is not conceivable for multiple source addresses from multiple networks (DDOS).

    The fact the server loads are massive indicates huge spawning of Apache children.
    It only indicates the inability for the server to handle the current requests, resulting in apache queueing the requests, which results in a high server load. 417 out of 437 processes sleeping more sounds like the inability to fork any additional apache processes. He probably had a run away apache process.

    While it IS possible he is receiving mass traffic, all things considered, the possibility of this causing huge server loads compared relatively to a DDoS is low.
    Agreed that it wasn't a problem with massive traffic, but it wasn't a DOS either. It was more than likely a malfunction.
    -Mark Adams
    www.bitserve.com - Secure Michigan web hosting for your business.
    Only host still offering a full money back uptime guarantee and prorated refunds.
    Offering advanced server management and security incident response!

  22. #22
    Join Date
    Nov 2000
    Location
    Austin, TX
    Posts
    415
    Next time (hopefully there won't be a next time) use CJCS's advice and look to see if its a certain site thats getting all the hits and look for any sort of pattern in the requesting IPs. Sometimes this can just be that someone had an endless loop or included a file that included itself etc into a loop, I've seen it before.
    Justin Bachus
    BlastHosting, LLC - Professional web hosting at a low price with
    PHP, MySQL, FreeBSD, and more!
    http://www.blasthosting.com

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •