Help - Apache Failing - Invalid method in request - will pay for help
I am having real problem with Apache. Its been down now for like 6 hours. I've tried fixing this with multiple re-installs (./easyapache), httpd.conf checking (manually), looking in /tmp /var/tmp directories. I've looked through the error_log and here's an example:
It is obvious he is being DOS'd, look at the requests! Unless it was pure luck that random letters were sent by a thousand different ips =|
Now, while we can't do a whole heap about the dosing, except for following the above links, check your httpd.conf to find the namevirtualhost. Then, under that find the respective virtualhosts for that namevirtualhost. If you have the namevirtualhosts written twice, delete one (merging the virtual hosts). If there is a namevirtualhost with nothing in it, delete that as well. Then restart apache.
Might be worth while taking the apache server offline while your playing with that config, doesn't sound like its doing much staying online =P
The quickest way i know to fix this would be to load the sites off of the ip that is being attacked, then close down requests to that ip (block them in iptables or APF). Or, if they are using the domain name to attack (possibly check the traffic coming through the DNS server to notify), then single out that domain, put it on its own seperate IP, then close down that IP. Im trying to track down some information on why this is happening considering its becoming very frequent around here. Until then, good luck
Ok I ran that command and it spat out the number: 9978
That would kind of imply its not an attack but something on the server. The rapid speed with which Apache reaches Max clients also makes me believe this is some kind of virus related thing. Plus in my /usr/local/apache/logs directory it keeps filling up with these ssl_mutex.xxxxx typ files that are all empty files - plus they are created by user "nobody". But there's nothing in /tmp /var/tmp
One thing you might try kind of tedious but is to go to WHM and suspend all the accounts on the server then try and restart apache. If it starts and will stay running then start unsuspending sites one at a time until it crashes that way you will know what site they are attacking.
I sound a wake up call to all out there who feel their servers are secure. Both chkrootkit and rkhunter both say the server is clean but I have discovered its been hacked. How do I know - because titles in WHM have been altered - there is a service in the Service Manager called "EvilApache" - its pretty clear whats happened. So we are in the process of migrating clients away from the server to another server. But I just am amazed by this attack, it was completely undetectable and nothing protected us from it - it seems to be exploting an Apache weakness of somekind. This is a serious issue that needs to be addressed fast. Now I am worried about all our servers. Anyone who's had this similar issue with Apache - its a Hack of somekind thats pretty damn stealthy.
So we are in the process of migrating clients away from the server to another server
Be careful what you migrate - carefully inspect every vhost's directory (especially for hidden files) before you migrate. Something might have gotten through one of the vhost's scripts, and you might migrate the 'implant' to a new server.
I just want to say to all who inputted to this thread I started that I am very grateful and thankful to have good people always willing to share advice and offer some help, it says a lot about what kind of people you are. Thank you all.
I have completed a OS Reload and it appears all has gone well (thank God), although I had a little shock. I noticed the "invalid request" errors started appearing again in the error_log, so I immediately disabled all shell access on the server - it stopped immediately - now the error_log looks perfectly normal again. So there's at least one way to deal with this issue and finding the culprit on the server before they do any real damage. Although I'm not 100% sure this stopped the problem (it could have been a coincidence) but no problems since disabbling ALL shell access on the server.