Results 1 to 39 of 39

Thread: HostDime is out

  1. #1
    Join Date
    Jul 2003
    Location
    Florida
    Posts
    54

    HostDime is out

    Sigh

    Apparently an APC hard reset the whole datacenter?

  2. #2
    Yeah thats odd 2 of my severs are up and my main one is down.

  3. #3
    There phone service is down too now mine is ringing.

  4. #4
    Join Date
    Jul 2003
    Location
    Florida
    Posts
    54
    I just talked to Miguel on the phone and he said there was a fire? He's not sure all the details yet but he was running into the NOC, and promised me a call back.

  5. #5
    Join Date
    Jul 2003
    Location
    Florida
    Posts
    54
    Manny is telling me that a transformer exploded in a nearby building, and people may be dead. The fire department made them shut down. Everything should be coming up now he says.

  6. #6
    Join Date
    Jun 2006
    Location
    London, Arizona, Utah
    Posts
    654
    Down from several locations still. I guess they will slowly bring it all back up!

  7. #7
    Join Date
    Feb 2006
    Location
    San Francisco, California
    Posts
    469
    Wow, hope for the best. What city is their DC located in?

  8. #8
    Join Date
    Jul 2003
    Location
    Florida
    Posts
    54

  9. #9
    Join Date
    Jul 2005
    Location
    In the Internets
    Posts
    3,622

  10. #10
    Join Date
    Jul 2003
    Location
    Florida
    Posts
    54
    Quote Originally Posted by larry2148 View Post
    Wow, hope for the best. What city is their DC located in?
    Orlando, Fl

  11. #11
    Join Date
    Jun 2002
    Location
    Orlando, FL
    Posts
    1,054
    Thanks for your patience, updates can be found here http://forums.hostdime.com/showthread.php?p=39031

  12. #12
    Down 4 hours and counting

  13. #13
    Still down here as well.

  14. #14
    Join Date
    Jul 2005
    Location
    In the Internets
    Posts
    3,622
    Why didn't the generators kick in, once you turned off power?

  15. #15
    We're back up here. Total service downtime = 4h24m. Last outage like this I experienced was May 2008 which lasted >6 hours.

  16. #16
    Just popped back up as well.

  17. #17
    Quote Originally Posted by nerdie View Post
    Why didn't the generators kick in, once you turned off power?
    Excellent question. Let's see if Mr. HostDime is as eager to answer this. Something doesn't quite add up with the "We had to shut down as a pre-cautionary measure" explanation insofar as some of their NOC racks remained up, powered, and accessable during the entire outage event.

    From http://forums.hostdime.com/showthread.php?t=7780 :
    How Long to the Real Cause of Today's Problems Comes Out?
    I understand there was an incident regarding the data center external power supply, but how does a facility with its own back up power generator and UPS, have a huge issue when their external power is disrupted, and how does Hostdime plan to prevent this occurrence in the future. Did someone forget to pay the fuel bill or not put in the proper disconnection switch to remove the data center from the grid?

    Iím running out of fingers on a hand to count the number of major power issues in the data center in less than two years.
    Last edited by evilelmo; 08-03-2009 at 08:00 PM.

  18. #18
    Join Date
    Dec 2001
    Location
    MO
    Posts
    629
    I asked about their backup generator in the past when this happened and I never received a response.

    Also, we're going on 12 hours now and a lot of my servers are still slow as hell. Their main site is currently down and the packet loss to ANY server in their datacenter is very, very high.

  19. #19
    Seeing that as well. Almost every refresh of my site results in "The connection was reset" Firefox error.

  20. #20
    Join Date
    Oct 2003
    Location
    Orlando, FL
    Posts
    244
    subigo,
    I was speaking with you on aim. The last report I got was "yep, they all look good again". Are you seeing issues again?

    Maupinf,
    If you are seeing issues. Can you send me a pm and I'll contact you personally?

  21. #21
    My server was down for more than 12 hours and submitted a few tickets without any resolution.

    What's going on with hostdime?

    Really a big disappointment seeing this happen on a data center and as wondering how come power generator backup never switch over.

    I had a big question mark on this ???

  22. #22
    Join Date
    May 2008
    Location
    Tampa, Florida
    Posts
    181
    wow that is crazy and what is even crazier is that we live about an hour from Orlando too so im surprised i didn't hear it on the local news.

  23. #23
    Paul, this morning things look good. No connective issues so far.

    Would be curious as well why the backup generators weren't used yesterday to keep the data center running if external power was cut/had to be cut.

  24. #24
    Join Date
    Jul 2003
    Location
    Florida
    Posts
    54
    Quote Originally Posted by nerdie View Post
    Why didn't the generators kick in, once you turned off power?
    They did kick in. There was a guy on a transformer so the fire department to be safe made the data center kill the generator in case there was back feed while attempting the rescue.

  25. #25
    Join Date
    Jun 2002
    Location
    Orlando, FL
    Posts
    1,054
    As Mix said above, a freak incident was a chain effect to something many didn't have control over. We are glad no one was hurt and the person, who was mentally ill didn't die. We sincerely apologize for any downtime you may experienced. I assure we had 25+ staff working around the clock to get things back to normal. We will happily apply any credit based on our SLA http://www.hostdime.com/policies/sla/ please file a ticker or a call so we can apply this for you.

  26. #26
    Join Date
    Nov 2005
    Posts
    1,224
    Quote Originally Posted by Mix View Post
    They did kick in. There was a guy on a transformer so the fire department to be safe made the data center kill the generator in case there was back feed while attempting the rescue.
    So does that mean anytime there are linemen working on the poles, such as repairing downed cables after a storm, the DC (and anyone else in the area with generators) will have to shut down?

    Something just doesn't smell right about that.

  27. #27
    Join Date
    May 2003
    Posts
    37
    Quote Originally Posted by HostDime View Post
    As Mix said above, a freak incident was a chain effect to something many didn't have control over.
    Why does it feel like -
    "teacher, the dog ate my homework."

  28. #28
    Join Date
    Jun 2002
    Location
    Orlando, FL
    Posts
    1,054
    Quote Originally Posted by Sekweta View Post
    So does that mean anytime there are linemen working on the poles, such as repairing downed cables after a storm, the DC (and anyone else in the area with generators) will have to shut down?

    Something just doesn't smell right about that.
    Perhaps you didn't read the events as they happened. We posted direct links from the media sources a long with videos. When the person from the mental institute was stuck between two transformers after being electrocuted the fired dept shut down and made everyone within the Plaza ( our building is within this plaza) shut down and evacuate as precautionary. The pictures posted taken on site show this as it happened.

    http://www.orlandosentinel.com/news/...,5160170.story

    http://www.wftv.com/video/20268586/index.html
    Last edited by HostDime; 08-04-2009 at 08:23 PM.

  29. #29
    Join Date
    Nov 2005
    Posts
    1,224
    Quote Originally Posted by HostDime View Post
    Perhaps you didn't read the events as they happened. We posted direct links from the media sources a long with videos.
    Actually, I did. Read the story, watched the video.

    Quote Originally Posted by HostDime View Post
    When the person from the mental institute was stuck after being electrified the fired dept shut down and made everyone within the 2 buildings next door shut down and evacuate.
    I asked that question because I've not heard of grid AND generator power being ordered OFF for a building except when there are environmental hazards such as gas leaks, flooding, etc.

    Power feedback onto the utility is a new one on me, especially for a facility that isolates itself from the grid when on generator power.

  30. #30
    HostDime - Since you are kind enough to reply here, can you give a direct answer to the question many of your customers are now asking:

    Why didn't the generators kick in, once you turned off power?

    Honestly, trying to get an answer to this question is like pulling teeth. Your customers are concerned - rightly concerned - that HostDime does not have a working system in place to deal with power outages. I remember when Hurricane Charlie hit Orlando in August 2004 and a caused a power outage at DimeNOC. Your generators did not work then either, and HostDime/SurpassHosting customers had to deal with a major and extended service outage as a result.

    Put yourself in your customers shoes. It appears (again) that you have no working backup power system in place to keep the servers functioning without hiccups when there's a power failure.

  31. #31
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    2,955
    How often do extended disruptions like this one happen at HD? Someone mentioned that a similar one took place in May. When did the previous disruption besides may happen and what is the duration? Also, some pointed out that their servers were kept up throughout. How come? Any answers to these questions is appreciated.

    Thanks!

  32. #32
    Join Date
    Jun 2002
    Location
    Orlando, FL
    Posts
    1,054
    Quote Originally Posted by evilelmo View Post
    HostDime - Since you are kind enough to reply here, can you give a direct answer to the question many of your customers are now asking:

    Why didn't the generators kick in, once you turned off power?

    Honestly, trying to get an answer to this question is like pulling teeth. Your customers are concerned - rightly concerned - that HostDime does not have a working system in place to deal with power outages. I remember when Hurricane Charlie hit Orlando in August 2004 and a caused a power outage at DimeNOC. Your generators did not work then either, and HostDime/SurpassHosting customers had to deal with a major and extended service outage as a result.

    Put yourself in your customers shoes. It appears (again) that you have no working backup power system in place to keep the servers functioning without hiccups when there's a power failure.
    evilelmo,

    There is actually a formal announcement on what happened being posted tonight to clients from a company official which is why nothing has been posted here yet. It was a priority to restore systems as priority as am sure you can appreciate. I do apologize the full explanation hasn't been released yet but it will be tonight.

    I am not sure if you realize this but we go on UPS/Generator about 2-3 times a month here half the year in florida due to the summer storms giving brown outs. Do you ever notice issue during those? Of course not because there is systems and place and surely you have seen the videos of it? http://www.hostdime.com/facilities/
    If indeed we didn't have a working system in place we would not be able to meet our 99.9% uptime each year which we have met every year since we been in business. No downtime is acceptable however it appears you are shinning on the times there is issues and I do not blame you. We are here to be nothing but perfect and we will continue to strive and do whatever it takes to keep everyone happy and push for 100% uptime year after year.

  33. #33
    Join Date
    Jun 2002
    Location
    Orlando, FL
    Posts
    1,054
    Quote Originally Posted by UNIXy View Post
    How often do extended disruptions like this one happen at HD? Someone mentioned that a similar one took place in May. When did the previous disruption besides may happen and what is the duration? Also, some pointed out that their servers were kept up throughout. How come? Any answers to these questions is appreciated.

    Thanks!
    Before this outage it was May 2008 approximately 15 months ago when we had a UPS battery malfunction. The duration of disruption has been less then 1 hour in most cases, although some servers experienced more due to the issues which came about when trying to bring them up. Everyone company has issues, how they are addressed is what makes the issue/mistake into something to be learned from and be better. We are continuously striving to nothing but perfect and we will continue to do this.

  34. #34
    Join Date
    Jul 2003
    Location
    Florida
    Posts
    54
    Guys,

    Without trying to look like a HostDime/DimeNOC cheerleader I was (probably/more than likely) affected a lot worse then most/all of you. I have A LOT of servers in dedicated cabinets with these guys, and I deployed my techs to the building at the same time.

    I have only been with these guys for literally a couple of weeks; but they showed a ton of professionalism, and gave us a very experienced Sr. Technician to use at our disposal to get our cabinets online. (Thank You Paul <3)

    I am with you guys in being upset about ANY down time but at the same time I would love to see any of you write a contingency plan that involves someone with a mental health disorder climbing a power pole.


    The main thing here is that it is VERY easy for us all to play Monday morning quarterbacks; but we were not the technicians that were on staff trying to make decisions while a guy was on the power pole being electrocuted. I think that we should all take a step back and be thankful that a fellow human even survived such an incident. I challenge you as a human being to watch the video - put yourself in the technicians place and be able to make such life changing decisions.

    At the end of the day it is my personal feeling whether you share it or not that a human life is far more important than a few minutes or even an hours worth of downtime.

    I shared these same thoughts with our customers and the response has been very overwhelming. Here is a link to our announcements page:

    http://www.made2own.com/clients/announcements.php?id=4
    Last edited by Mix; 08-05-2009 at 03:24 AM.

  35. #35
    Join Date
    Nov 2005
    Posts
    1,224
    Quote Originally Posted by HostDime View Post
    the fired dept shut down and made everyone within the Plaza ( our building is within this plaza) shut down and evacuate as precautionary.
    I'm having problems with this statement. I'm not saying it isn't true becasue I wasn't there. It does make sense that they would cut utility power to rescue someone from amongst the power lines.

    What does not add up is the alleged mandatory evacuation of all buildings in the plaza. It was a power outage, not a gas leak with risk of explosion. If anything, they would want people to stay INSIDE the buildings to keep the area clear of spectators.

    In fact, that is precisely what was said in the news article-- "Police and fire workers cleared observers out of a rear parking lot at the request of Lakeside Behavioral Healthcare administrators before the final rescue effort began."

    [remove spectators from parking lot] != [evacuate buildings]

  36. #36
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    So any time someone wants to shut down hostdime they climb the utility pole outside the building and sit there? Seems like a major security problem any way you look at it.

  37. #37
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    2,955
    Quote Originally Posted by HostDime View Post
    Before this outage it was May 2008 approximately 15 months ago when we had a UPS battery malfunction. The duration of disruption has been less then 1 hour in most cases, although some servers experienced more due to the issues which came about when trying to bring them up. Everyone company has issues, how they are addressed is what makes the issue/mistake into something to be learned from and be better. We are continuously striving to nothing but perfect and we will continue to do this.
    Thanks for taking the time to answer my questions. I hope all is back to normal now. Frankly, I would rather co-locate at a facility that is prepared for storms and flooding as opposed to one that isn't well prepared because it thinks its FEMA 500-years flood plain is safe.

    Regards

  38. #38
    Join Date
    Jun 2002
    Location
    Orlando, FL
    Posts
    1,054
    Quote Originally Posted by Sekweta View Post
    I'm having problems with this statement. I'm not saying it isn't true becasue I wasn't there. It does make sense that they would cut utility power to rescue someone from amongst the power lines.

    What does not add up is the alleged mandatory evacuation of all buildings in the plaza. It was a power outage, not a gas leak with risk of explosion. If anything, they would want people to stay INSIDE the buildings to keep the area clear of spectators.

    In fact, that is precisely what was said in the news article-- "Police and fire workers cleared observers out of a rear parking lot at the request of Lakeside Behavioral Healthcare administrators before the final rescue effort began."

    [remove spectators from parking lot] != [evacuate buildings]
    Perhaps this image http://www.hostdime.com/images/Orlando.png will help you understand more. Evacuation was NEVER mandatory but we were advised of what was being done and as a precaution and for the well being of everyone we had over 50% of our staff evacuated and only kept key people in the facility. The proximity of the main issue where this guy was at was so close, as you can see in the picture from an aerial view!, it was pretty close that if an explosion with those two transformers could have caused some serious damage. The newspaper isn't aware of everything that is done or said in these type of media as you know. Perhaps if you were here you would understand what transpired easier.

  39. #39
    Join Date
    Nov 2005
    Posts
    1,224
    If the concern was the risk of transformer explosion, then it's making a lot more sense now (except why would they be worried about that after the power was cut). Since I was not there, the only information is that which comes from WHT posts, media coverage, and "this doesn't immediately make sense" speculation.

    I also agree that authorities often over-react to circumstances out of what they consider an abundance of caution. How often have we seen an interstate highway shut down in both directions after a serious accident, even though the crash was contained to one side, separated by a 30+ foot median? IMO, the risk of transformer explosion was essentially negated when power was cut.

    I'm glad to hear your ops are back to normal, and am sorry anytime I see folks inconvenienced and/or incur financial loss because of some criminal, careless moron, or (in this case) mentally ill person, over whom you had no control. It stinks even more if you had the ability to cover the power outage via on-site generator, and weren't permitted to use it.

Similar Threads

  1. Hostdime Out?
    By DeltaAnime in forum Providers and Network Outages and Updates
    Replies: 7
    Last Post: 12-08-2004, 03:20 AM
  2. hostdime.com
    By dany0402 in forum Web Hosting
    Replies: 34
    Last Post: 11-23-2004, 11:31 PM
  3. What you think about hostdime.com
    By spv78 in forum Reseller Hosting
    Replies: 9
    Last Post: 11-12-2004, 05:36 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •