Results 1 to 18 of 18
  1. #1
    Join Date
    Nov 2001
    Location
    The South
    Posts
    5,403

    Only one server left with EV1

    Well I just canceled one of my last two servers with EV1, and here's why:

    You said order a restore for this server, before I had a chance to properly track down the php script running to fix the problem. Your ddos protection (is very good, kudos) picked up the errant process in SECONDS, way before I could even shell in and look at what's happening! So guess I can't do much about that when the server goes offline even before my system emails me to let me know a process was launched by apache! That's fast, too bad it's self defeating.

    A restore will fix nothing, when the customer puts the same scripts on that server without knowing WHICH SCRIPT is being exploited to launch processes it'll just happen again.

    I implemented extra logging and an auto-kill crontab running every MINUTE so I could catch it while it was happening and properly diagnose which script is being used to start up this process, but you took the server offline faster than I could even HOPE to react to the process launching.

    Fact of the matter is, if I'm gonna rebuild this server, which you're forcing me to do, I'm gonna rebuild it at GNAX where I have a more 1 on 1 communication with the staff and will be allowed a chance to fix the problem without an UNNECESSARY and UNNEEDED restore that won't fix a darn thing. If worse comes to worse I can just drive over there and grab the server myself and start forensics on it (no forensics needed, this isn't a rooted server, it's a php script launching a process that dos's an irc server, all I have to diagnose is what php script is doing it, then that's that, it's fixed).

    ALso a good chance to once again reduce my bill here, thanks. Now I only have a single server with EV1, down from a high of 12 or 14.
    So what was this all about?

    Some php process was being exploited on this server to launch a program that attacks an irc server, pretty standard script kiddie stuff.

    I got the first notice yesterday out of the blue, when I got access back I logged in, couldn't determine anything from the httpd logs (probably "POST" instead of "GET" which totally makes apache logs useless) so I started running a "top" and a minutely cron to see what apache owns in the process list. This customer has PHPBB so that's the first thing I checked - it was version 2.0.17 the latest version as far as I know of so maybe that's not it, so until I -catch it red handed- (via logging) I can't know which php program he's runnign that's the bad guy.

    Woke up this morning the server was down again, another attack, well I couldn't login to look at my logs, when I logged in the logging apparently wasn't enough, so I upped the logging to a much much more extreme level, and also implemented a cron that ran every -minute- that:

    A: looked for apache owned processes that weren't proper HTTPD processes
    B: logged the process that wasn't httpd, pid/parent pid/etc
    C: logged a snapshot of the /server-status output so I can see the parent PID of the process launching the bad process and what URL exactly launched it
    D: killed the non httpd process so it won't run

    Well an errant process started and -5 seconds- later the server was offline.

    Ok fast response to that crap is GOOD kudos to EV1 on some EXTREMELY good network setups there.

    But I can't be expected to react to something in 5 seconds! Now the server is offline so of course I can't fix it, I can't login and look at my logs, and I have to order a restore. Problem is the logs that should show me WHAT SCRIPT TO TELL MY CUSTOMER TO UPGRADE are ON THE SERVER about to be deleted, so that does me NO GOOD, and the second this web site is "restored" it's gonna STILL BE VULNERABLE.

    Sorry, I got empty servers in my colocation at GNAX and if I have a problem like this with GNAX they know I'll fix it and allow me to fix it, so that leaves me with only a single server at EV1.

    Guess I prefer to have a name, not a number when I deal with my providers. It's awesome that EV1's dos protection is that fast, too bad it's so fast I can't even respond to the problem.
    Gary Harris - the artist formerly known as Dixiesys
    resident grumpy redneck

  2. #2
    Join Date
    Apr 2005
    Posts
    175
    Why not have them bring it back online but configure your firewall to only allow SSH, in and out. To allow you to have it up without it dossing anything?

  3. #3
    Join Date
    Nov 2001
    Location
    The South
    Posts
    5,403
    Because "3 aup violations in 1 day = restore" and that's that.

    I ran a backup this morning, I had a feeling this was gonna happen, when I saw how fast they were taking that server offline when the process launched I knew I had no chance to catch this in action and do anything about it, I also know EV1 is pretty inflexible about these things, I have spare servers in my cage at GNAX, I'll have another server online quicker than I could bitch and moan my way through the EV1 support heirarchy and it saves me some money each month by allowing me to further reduce my rented server pool so while it's a pain in the butt, it's not gonna raise my blood pressure either, gotta roll with the punches.
    Gary Harris - the artist formerly known as Dixiesys
    resident grumpy redneck

  4. #4
    Join Date
    Mar 2001
    Posts
    1,434
    Glad you had a backup plan in place to get around this. That's the problem with hosting companies running their business under these faceless giants such as EV1. When there's a problem, you are server #3,475, and get the same treatment as the 11 year old running a game server.

    The smaller DC with a personal touch is the way to go for a "small" hosting company who leases / co-locates servers, as you can develop a working relationship that benefits both parties as you grow. I cannot imagine having servers with a DC that does not offer a way to communicate directly and effectively with a high level manager during times like these.

    - John C.

  5. #5
    Join Date
    Apr 2005
    Posts
    175
    Yeah I could see how they might have such a policy.

    Sounds like a good plan you have.

  6. #6
    Join Date
    Jun 2003
    Location
    United States of America
    Posts
    1,838
    that communication with the dc is pretty important and i guess knowing what there capable of is too thanks for the heads up buddy
    Computer Steroids - Full service website development solutions since 2001.
    (612)234-2768 - Locally owned and operated in the Minneapolis, Minnesota area.

  7. #7
    Join Date
    Nov 2001
    Posts
    5,383
    Hah we had the same problem when we left them 2+ years ago. SQL inject on a script, they took down the server a few minutes after it happened and said your box was compromised and order a restore. So while they were fiddling their thumbs restoring the server I asked them to put the drive in as a slave, what do you know it was not a root exploit. They hire people that have no idea.
    Clustered Hosting With Continuous Data Protection (CDP)
    http://www.solidinternet.com
    8 Years of hosting excellence!

  8. #8
    Join Date
    Feb 2003
    Location
    Kuala Lumpur, Malaysia
    Posts
    4,974
    I think none of the (cheaper ones) big providers could provide top notch service eh?

  9. #9
    Did you emailed the headsurfer this?
    ^_^

  10. #10
    Join Date
    Nov 2001
    Location
    The South
    Posts
    5,403
    Originally posted by Francisco
    Did you emailed the headsurfer this?
    Why would I do that? Even if I thought it would help, that's not really the point, the point is be very aware of the way things work and make sure you're not caught off guard if it happens to you. The simple fact is EV1 is too big to go spending a lotta time dealing with me or you or anyone on these issues in anything but a cookie cutter manner, it sucks for the little guy, but they're not gonna assign one of their "upper echelon" security guys to spend ANY time trying to track down this peon problem "restore" done, problem solved on their end.

    It's a matter of logistics, they can't help you or me or anyone track down a vulnerable php script "restore" is the answer the question isn't really relevant on their scale.

    Yeah maybe I could email Robert or Mario or someone but that isn't the point, the point is be VERY AWARE that if your server suffers 3 aup violations in one day this can happen to you, make backups, don't bitch when your server goes offline and you complain about how you ain't made a backup in months and you're losing months of changes etc etc (I say "you" I mean "anyone" not you).

    I've made the comment before how I've never had a server pulled and I've always answered any aup violation notices promptly and well, now it's happened to me despite the fact I replied promptly and did all I could do as quickly as possible to find and fix the problem, so I guess somewhere someone is pointing at me and saying "Ha Ha" (think: The Simpsons)
    Gary Harris - the artist formerly known as Dixiesys
    resident grumpy redneck

  11. #11
    Join Date
    Apr 2002
    Posts
    447
    I think you will be happy at GNAX, I have used them for a long time now and their support is very very good, they always do whatever they can to work with you to solve a problem.

  12. #12
    A bit OT, I know, but how happy are you with GNAX? (customer service)

  13. #13
    Join Date
    May 2002
    Location
    singapore
    Posts
    455
    I am just about to press the order button at ev1. Thanks, Dix for letting me know of such issue.
    current and satisfied customer of softlayer.com and webnx.com

  14. #14
    Join Date
    Jan 2002
    Posts
    1,033
    If I remember correctly, I ditched my last EV1 box about two years ago due to horrid/inflexible customer service.

  15. #15
    Join Date
    Sep 2002
    Location
    Toronto, ON
    Posts
    3,439
    I must say ever since the SCO settlement I never really considered EV1. The experience you had with them Dixie just gives me an other reason to stay away from them, as an admin myself I cannot imagine a company forcing me to restore without having a chance to track down the attacker and the source of the problem.

    You don't scrap a car because it has a flat tire.
    Jean-Pierre Abboud / I'm the TekGURU
    www.Gotekky.com / Managed and Self-Managed hosting solutions
    Toll free: 1.888.915.4400 / Local: 1.514.316.1885 / Live chat
    Cloud VPS Hosting

  16. #16
    Join Date
    Nov 2001
    Location
    The South
    Posts
    5,403
    Originally posted by Walter
    A bit OT, I know, but how happy are you with GNAX? (customer service)
    GNAX? They're great, I've used them since the earliest days of my hosting business, my 4th server was a GNAX dedicated (that I later bought from them and had it moved into my cage that I eventually upgraded to there).

    I now have all, every one, of my hosting servers at GNAX, I now only have 3 servers outside of GNAX - the server that hosts MY web sites and related support site (helpdesk/billing/forums) so that it is outside of gnax in the event there is a network issue with them my forums/etc are still online. I have a server still at EV1 that hosts my private irc server for customer support, and I have a server at .. 800hosting? forget the exact url, it has my mrtg setup, "in house" monitoring, and a few other services running that are better to have outside of the network you're monitoring.

    Customer service has always been fine, keep in mind GNAX is small, so if things get busy over there, a new virus bogging networks, a new phpbb exploit that's causing a LOT of bindshell and zombie attacks, that sorta thing, they can get a bit slow on responding to basic requests, they're not a huge company with 100's of people employed, they're smaller, which is a good thing (personal service, you're a customer not a number) and a can be a bit of a bad thing but I'd MUCH rather take the few "bad sides" to a small company in order to have the good, as to take the bad of a big company (impersonal service, support by numbers, inflexible responses to problems that occur).
    Gary Harris - the artist formerly known as Dixiesys
    resident grumpy redneck

  17. #17
    Dixiesys, thank you very much.
    Did you go trough one of their resellers or directly through TranXact Global ?

  18. #18
    Join Date
    Nov 2001
    Location
    The South
    Posts
    5,403
    Originally posted by Walter
    Dixiesys, thank you very much.
    Did you go trough one of their resellers or directly through TranXact Global ?
    I've always dealt directly with GNAX/Tranxact Global
    Gary Harris - the artist formerly known as Dixiesys
    resident grumpy redneck

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •