Results 1 to 30 of 30
  1. #1
    Join Date
    Jul 2004
    Posts
    868

    Angry Apache Crashed !

    Every time when I restart apache in WHM I have error:
    httpd has failed, please contact the sysadmin.

    i recompiled apache but nothing ! it worked only for few seconds and ...

    Here is few lines from apache error log:
    tail -n 200 /usr/local/apache/logs/error_log | more

    Code:
    [email protected] [/usr/local/apache/logs]# tail error_log --line=100
    [Thu Jul 28 16:08:10 2005] [error] [client 68.237.172.248] Invalid method in request tyQfLH6khlyhaurlgDjkJC
    [Thu Jul 28 16:08:11 2005] [error] [client 83.79.140.27] Invalid method in request e9T4WJ
    [Thu Jul 28 16:08:13 2005] [error] [client 80.78.132.11] File does not exist: /home/linkdoni/public_html/logo/logo.gif
    [Thu Jul 28 16:08:13 2005] [error] [client 80.78.132.11] File does not exist: /home/linkdoni/public_html/404.shtml
    [Thu Jul 28 16:08:19 2005] [error] [client 80.253.131.2] File does not exist: /home/linkdoni/public_html/_vti_bin/owssvr.dll
    [Thu Jul 28 16:08:19 2005] [error] [client 80.253.131.2] File does not exist: /home/linkdoni/public_html/404.shtml
    [Thu Jul 28 16:08:25 2005] [error] [client 66.136.185.94] Invalid method in request 8plUTtnJXGpGBl7QmliSUrtNUZrQJuuxBSEJgeLyOhKPfJp5LyHGoYA5fw8kNybhOHOfFAWMmRS3ubt1sSq2
    [Thu Jul 28 16:08:25 2005] [error] [client 66.136.185.94] Invalid method in request aBDnfFaipKv
    [Thu Jul 28 16:08:32 2005] [error] [client 201.235.69.58] Invalid method in request VxMs3bNzTJjmPLxlcdYejyPDRBujWn7T7474tYf5l1qb6jz8bEtu
    [Thu Jul 28 16:08:32 2005] [error] [client 201.235.69.58] Invalid method in request vjE5RaBYByEfpiJaLD8WBQfVf6TNkzue0K4ickKFziG2zQVR
    [Thu Jul 28 16:08:32 2005] [error] [client 211.244.222.87] Invalid method in request tcjNA0boOOA4ePjIM
    [Thu Jul 28 16:08:33 2005] [error] [client 68.215.78.126] Invalid method in request cWBlyoBvHTgeJhdVhqBDgLDyl0j8GxOHEFTnZ
    [Thu Jul 28 16:08:33 2005] [error] [client 68.215.78.126] Invalid method in request zT8ngpxYI43OrshOj5RU9qT3eSPXjAkdUj
    [Thu Jul 28 16:08:56 2005] [error] [client 218.226.122.248] Invalid method in request z7X8vluxxwh8NnJYF23XqAtWymPVZuLW9AcE5f87E6c
    [Thu Jul 28 16:08:56 2005] [error] [client 218.226.122.248] Invalid method in request 2u5y4Rpc0KmdsFpjmT6Me7hT0CmGekgjY8Zs6UCezk4
    [Thu Jul 28 16:09:05 2005] [error] [client 201.235.18.163] Invalid method in request YoxN7B3E5XO2wjG8
    [Thu Jul 28 16:09:05 2005] [error] [client 201.235.18.163] Invalid method in request qKZtPNaIh4UhrZDaX8r35f18Fl7raskf4jXobkrL13KoamcJDRSwp6x8r9JvjOVan1Ap7Yw
    [Thu Jul 28 16:09:10 2005] [error] [client 201.235.18.163] Invalid method in request
    [Thu Jul 28 16:09:10 2005] [error] [client 218.47.183.115] Invalid method in request 7cXRVQrdcBCPV5znyzLmF7ShgPUZ8min4ZYUTlKfX2I8lAFHBDsg1P5iRLVx3zVJfuCvdsNfhEVczPS4TL2tZ8dVx59P3FiHCS
    [Thu Jul 28 16:09:10 2005] [error] [client 201.235.18.163] Invalid method in request HG0wvXmRe71Yd63M9hQJaWN504fgxJyA2xUI47qCPZhvwlITOBnSNU92W3YyjC9Fcz
    [Thu Jul 28 16:09:10 2005] [error] [client 218.47.183.115] Invalid method in request Z8Cqq6sRydmlSueW5UvKqTUxNrKXoZLDIdJfcz63F6Er0Ql4e9mOL3dYzCIX
    [Thu Jul 28 16:09:11 2005] [error] [client 65.0.101.130] Invalid method in request d1BvsPQNBeFzKYgLrk3NfqRUqoxPl0MHUf1lEPzjAfmgIRx2rCECUlBzzUzJfnhZyZO
    [Thu Jul 28 16:09:11 2005] [error] [client 65.0.101.130] Invalid method in request rbj0M7QmDVxHyVXYsJbRfpUsDvuPGnYQvfS
    [Thu Jul 28 16:09:24 2005] [error] [client 60.236.204.111] Invalid method in request 7YgJdd8ZCtcsYiZKWAjy9hkPJLtEsmoNXulgMs4sIaRtZmU5mKQdjVV9NlgQAiFfGjC1hq36w90n
    [Thu Jul 28 16:09:26 2005] [error] [client 200.218.19.18] Invalid method in request frtqje6swSNAcrmhvuKxVBsmiNTxf19X4
    [Thu Jul 28 16:09:26 2005] [error] [client 200.218.19.18] Invalid method in request wFm6NBbnyfgnfLZTmINW2SYcx4cLE9HJGTuawIpKtCGxvHK0ihmPQeb2vcP33aXWBw8Rvj98i3S
    [Thu Jul 28 16:09:33 2005] [error] [client 195.175.37.6] File does not exist: /home/linkdoni/public_html/logo/logo.gif
    [Thu Jul 28 16:09:33 2005] [error] [client 195.175.37.6] File does not exist: /home/linkdoni/public_html/404.shtml
    [Thu Jul 28 16:09:57 2005] [error] [client 195.157.235.151] Invalid method in request yyT0XA83QCeoroxDNTN8JyLB
    [Thu Jul 28 16:09:57 2005] [error] [client 195.157.235.151] Invalid method in request 2PH
    [Thu Jul 28 16:10:06 2005] [error] [client 65.10.2.68] Invalid method in request cXFiSgtOwI4yhy9

  2. #2
    Join Date
    Jul 2001
    Location
    Singapore
    Posts
    1,790
    Run below command to check for syntax error:
    Code:
    /usr/local/apache/bin/apachectl configtest
    Giam Teck Choon
    :: Join choon.net Community today to share your tips and tricks on server issues please ::
    :: Singapore Dedicated Servers :: Singapore Virtual Private Servers :: Linux/FreeBSD Server Management ::

  3. #3
    Join Date
    Jul 2004
    Posts
    868
    yes i did
    but showing syntax is ok

  4. #4
    Join Date
    Jul 2004
    Posts
    868
    okey now it working after recompiling apache
    but there is alot of likes like this :
    31-0 2361 0/2/2 R 0.01 98 0 0.0 0.00 0.00 ? ? ..reading..
    32-0 2362 0/2/2 R 0.01 76 0 0.0 0.00 0.00 ? ? ..reading..
    33-0 2363 0/5/5 R 0.01 20 0 0.0 0.04 0.04 ? ? ..reading..
    34-0 2364 0/3/3 R 0.01 3 10399 0.0 0.04 0.04 ? ? ..reading..

  5. #5
    Join Date
    Apr 2005
    Location
    India
    Posts
    20
    Please have a check on the apache error_logs and also check whether you are under dos attack. May be too many requests than the apache can handle.

    ----
    Aby,
    Kekays
    Aim: kekays
    Yahoo: kekays
    Msn: [email protected]

  6. #6
    Join Date
    Jul 2004
    Posts
    868
    i dont know
    there is only 1mbit trrafic and its normal !

    in apache server-status

    there is a lof request like this :
    0/0/26 . 0.00 101 0 0.0 0.00 0.08 66.98.0.32 localhost.localdomain CsLiFW9UlzM8rJ8tcc4cq0Yp63fP3YPYXiXRqEAwPiHy8
    62-1 - 0/0/24 . 0.04 110 0 0.0 0.00 0.21 218.227.52.72 (unavailable) yFO


    and like this :
    37-1 5523 0/14/47 R 0.14 24 4087 0.0 0.13 0.36 ? ? ..reading..
    38-1 5438 0/5/44 R 0.03 18 6658 0.0 0.19 0.51 ? ? ..reading..
    39-1 5257 0/20/53 R 0.27 5 4261 0.0 0.17 0.65 ? ? ..reading..
    40-1 5712 0/16/90 R 0.08 85 5 0.0 0.10 0.57 ? ? ..reading..


    now cpu load is normal ( 0.05) and memory usage low

    but i am worried about that proccess !

  7. #7
    I would say you are under a ddos attack. How many open apache servers are there? I had a problem like this a little while ago with one of my servers. Got the same error, then in shell I typed killall apache then I could restart apache, but then moments later it was crashed. If you find it to be a ddos attack contact your datacenter.
    Eleven2 Web Hosting - World-Wide Hosting, Done Right!

  8. #8
    Join Date
    Jul 2004
    Posts
    868
    seems again i am under attack

    but i am unable to find ip's

    becouse there hounders of ip's that sedning that type of requests !

    what can i do ?

  9. #9
    Join Date
    Jul 2004
    Posts
    868
    [email protected] [/usr/local/apache/logs]# cat /proc/net/ip_conntrack | wc -l
    42348

  10. #10
    Join Date
    Jun 2005
    Location
    India
    Posts
    123
    Hello,

    Type the following command and paste the result here

    ----------------------------------------------------------------------
    netstat -apn | grep :80 |wc -l

    and

    netstat -apn | grep SYN_RE | wc -l

    ----------------------------------------------------------------------

    With regards,
    Bijo.
    Last edited by bijo; 07-30-2005 at 08:07 AM.

  11. #11
    Join Date
    Jul 2004
    Posts
    868
    [email protected] [/usr/local/apache/logs]# netstat -apn | grep :80 |wc -l
    756


    [email protected] [/usr/local/apache/logs]# netstat -apn | grep SYN_RE | wc -l
    386



    again under same attack !!

  12. #12
    Join Date
    Jul 2004
    Posts
    868
    after blocking more than 400ip !



    [email protected] [/usr/local/apache/logs]# netstat -apn | grep :80 |wc -l
    1459
    [email protected] [/usr/local/apache/logs]# netstat -apn | grep SYN_RE | wc -l
    718


    seems they have unlimited ip's

  13. #13
    Join Date
    Jul 2004
    Posts
    868
    last report :

    [email protected] [/usr/local/apache/logs]# netstat -apn | grep :80 |wc -l
    898
    [email protected] [/usr/local/apache/logs]# netstat -apn | grep SYN_RE | wc -l
    919



    still blocking ip's .....

  14. #14
    Join Date
    Apr 2004
    Location
    Australia
    Posts
    419
    I really wouldn't bother trying to block that many ips, you may slow it down a tad but not enough to be worth the effort. The only effective way so far that i have found is to move your sites to a different IP. Now, most of the time attackers use domain names to locate your ip, the other percentage is IP direct attacks. What you will need to do is transfer your accounts over and see which account takes the attack with it. When you have found it, put it on a different IP altogether and then block access to port 80 on that IP.

    Please post output of this command (let it run for about 20 seconds)

    tcpdump port 80

    Thanks

  15. #15
    Join Date
    Jul 2004
    Posts
    868
    >Matt -Seeksadmin

    thank you for your idea , now i am testing it

    how much that tcpdump will take ? this is more than 5 minutes and still working ...

  16. #16
    Join Date
    Jul 2004
    Posts
    868
    okey
    this is last dcpdump reslut
    Attached Files Attached Files

  17. #17
    Join Date
    Apr 2004
    Location
    Australia
    Posts
    419
    Artin - best to go with what i originally said and move your accounts to another IP. Im currently looking into a way to block it using IPtables or a mod to apache, but that could be a long time waiting. Thanks for the tcpdump.

  18. #18
    Join Date
    Jul 2004
    Posts
    868
    still under attack

    more than 2000 ip blocked but nothing

  19. #19
    Join Date
    Jul 2004
    Posts
    868
    i tried this command but this isnt blocking SYN request
    which command i should add in following line to block syn requests ?

    216.216.216.216 is server main ip

    iptables -A INPUT -p tcp -s 0/0 -d 216.216.216.216 --dport 80 -j DROP

  20. #20
    Originally posted by artin1982
    seems again i am under attack

    but i am unable to find ip's

    becouse there hounders of ip's that sedning that type of requests !

    what can i do ?

    Artin take a look at Top Layer Hardware for protecting against DDoS attacks. It is inexpensive and works - keeps your business going. Many Web Hosters and web sites use it.

  21. #21
    Join Date
    Jul 2004
    Posts
    868
    >abhayjoshi

    thanks i already sent request to DataCenter and waiting for they answer,

    and for now i just wana do what Matt -Seeksadmin said ,

    i moved all sites from main shared to other ip and i used following commands

    iptables -A INPUT -p tcp -s 0/0 -d 216.216.216.216 --dport 80 -j DROP

    but still i have SYN request on this ip , i think i should add more options to this command

  22. #22
    Join Date
    Jul 2004
    Posts
    868
    nothing ?

    plz someone help me

  23. #23
    Originally posted by artin1982
    nothing ?

    plz someone help me

    Artin...you really may want to consider Top Layer. They fit in most budgets. Let me know if you want me to call you and discuss.

  24. #24
    Join Date
    Mar 2002
    Location
    St. Louis, MO
    Posts
    1,378
    What control panel are you using if any? just curious

  25. #25
    Join Date
    Jul 2004
    Posts
    868
    cpanel
    centos 3.4

  26. #26
    Join Date
    Mar 2002
    Location
    St. Louis, MO
    Posts
    1,378
    Do you have a spare IP? If nothing else Re-IP your entire server, you can use Cpanel to do this... itll stop the dos attack, for the moment till you can get something under control or worked out. Otherwise bring up the apache status page in WHM, see where the majority of traffic is going...

    If it was one particular site, Id move them off the server, re-ip the entire server, go from there.. otherwise you wont have any choice but to invest in a firewall that has NIDS dection or Syn filtering, something along that line

    or... as a last resort, block the entire gateway the attack is coming from.... you use at your own risk though

  27. #27
    Join Date
    Jul 2004
    Posts
    868
    okey great news

    i am unable to ssh in to server

    what is next step ?

  28. #28
    Join Date
    Mar 2002
    Location
    St. Louis, MO
    Posts
    1,378
    Restart it from WHM see what happens... if your server is saturated from the attack, and cant login via SSH, have it rebooted, log in quickly... start doing some admining...

    If you are unable to admin your box, contact your provider and talk with them over it or hire an admin that can help you out. Dos attacks are like a virus, just dont simply go away... :-|


    Just to add (Edit in) If you have the server re-IPed, the old IP you will have to remove from your server, notify your provider of what happened
    Last edited by DigiCrime; 08-02-2005 at 03:38 PM.

  29. #29
    Join Date
    Jul 2004
    Posts
    868
    thanks for your idea but i am unable to connect with whm too !

    and for ip , its my main shared ip ! not a normal ip

  30. #30
    Join Date
    Mar 2002
    Location
    St. Louis, MO
    Posts
    1,378
    ouch... im assuming you are with Layer'd Tech assuming the site in your sig is one the server being attacked. Have you talked to them about anything yet ?

    only two options at this point, Re-IP or wait out the dos attack which I wouldnt recommend if its a fairly significant attack...

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •