Page 3 of 3 FirstFirst 123
Results 51 to 73 of 73
  1. #51
    Join Date
    Jan 2005
    Location
    Europe
    Posts
    59
    Well, I assume I'll be the first one daring to questionise this, but I can't say pretty much anything else that it's a pretty weak and badly planned datacenter, if there's no lightning rods/protectors on the roof of the building, no matter how low or tall the building is. It's just a basic safety measure, just like fire suppression is. Because if the lightning bolt would have been diverted to ground with the help of the lightning protector, the damages would have been next to none. Or if there was lightning protection, how was the groundwire routed and in what shape was it?

  2. #52
    Quote Originally Posted by FredForrest
    Actually if they took a direct strike and got things back up as fast as they did, the electrical system as well as network infrastructure is very well protected. The fact that most servers appear to be back up and not fried after the building took a direct hit indicates some pretty good protection in place.
    Our power company has confirmed that we did take a direct lightning strike. The fuses up on the pole are blown, and they are in the process of testing the cables before replacing the fuses. We're on generator right now and have plenty of fuel.

    We've got every staff member that we could contact called in and rebooting/fixing servers that didn't come up on their own.

    We also have a problem with our phone system and are working on restoring normal service.

  3. #53
    My server is now online, but I have a problem. APF is blocking smtp. Does anyone know what could be a problem?

  4. #54
    Join Date
    Jan 2004
    Location
    Sydney
    Posts
    148
    I had to restart mail servers after my server came up - dont know if that helps

  5. #55
    Join Date
    Jun 2003
    Location
    Rocky Mount
    Posts
    27
    Quote Originally Posted by masssa
    My server is now online, but I have a problem. APF is blocking smtp. Does anyone know what could be a problem?
    SMTP problems are all we're seeing right now too...EXIM seems to be having a fit every once in awhile. Hopefully, they'll have it all cleared up soon. We need to send mail.

  6. #56
    Is this the same issue affecting Site5 servers? They are saying a MI facility is down. But my server isn't up yet.

  7. #57
    Server have been running extremly slow after its back. Anyone experiencing the same thing?

  8. #58
    Hmm, so Site5 reseller and LiquidWeb host in the same facility. I was thinking of switching to LiquidWeb because of all the downtime I've been experiencing lately, but that would've been a mistake.

  9. #59
    To Ross / STL

    Quote Originally Posted by RossH
    Why would you pay an SLA credit on an "act of god"?
    Act of God? What Act of God... A lightning, storm etc isn't an act of God, you can prepare yourself for that type of situation (eventhough is rather unlikely it does happens)

    A meteorite falling on your head IS an act of God however...

    Hope our data is fine though...

  10. #60
    Join Date
    Jun 2003
    Location
    Istanbul
    Posts
    662
    Okay,
    No need to panic like that i believe, let them work on the issues calmly.
    I am pretty sure liquidweb is taking care of the any problems as human can do now.
    We will be past this issue with 2-3 hours of outgate only.

  11. #61
    Quote Originally Posted by Burnspot
    SMTP problems are all we're seeing right now too...EXIM seems to be having a fit every once in awhile. Hopefully, they'll have it all cleared up soon. We need to send mail.
    I solved mine this way:

    /sbin/service syslog restart
    /sbin/service exim restart

    In one of my boxes, it worked right out. On other box, I had to reboot the server. The antirelayd was failing due to /var/log/maillog rotation problems (runing Fedora/CPanel on the boxes)

    Got the tip on CPanel's forum:

    http://forums.cpanel.net/showpost.ph...56&postcount=4

  12. #62
    Join Date
    Apr 2006
    Location
    italy
    Posts
    0
    Quote Originally Posted by LiquidWebJer
    We will be paying an SLA credit to anyone that would like it.

    More information will be posted as it becomes available.
    Hi Jer,

    where i can find more information about this refound?

    i don't wanna speculate, i love LW support, but i have over 100 clients very very bad :°°

    the server down was append in italy, during the most important part of the day.

    thanks a lot.

    matteo.

  13. #63
    Quote Originally Posted by matteo
    Hi Jer,

    where i can find more information about this refound?

    i don't wanna speculate, i love LW support, but i have over 100 clients very very bad :°°

    the server down was append in italy, during the most important part of the day.

    thanks a lot.

    matteo.
    I too would like to know more about this.

  14. #64
    Join Date
    Nov 2005
    Posts
    223
    liquidweb is not down now; I enter well to their place web.

  15. #65
    To me (granted, I am an outside observer) it seems that this was handled phenomenally well, and I say kudos once again to LiquidWeb. It looks like they were down (for the most part) for only about 20 minutes after taking a direct lightning hit. I would say the people that are saying that a better design would have kept this from happening are wrong. Good job, LiquidWeb.

  16. #66
    Join Date
    Jun 2007
    Posts
    82

    nice recovery, but...

    It sounds like the systems were brought back online as quickly as possible and that LiquidWeb has been nice about communicating such here on this board.

    However,

    This is proof positive that LiquidWeb does not have the infrastructure in place to support hosting of revenue-generating sites or mission-critical sites that need to be up 24x7. The Italy post above is just one example. To argue otherwise is silly: lighting = downtime = not good.

    If you can deal with the occasional outage LiquidWeb seems fine. If you really need uptime, you need to go with a better host. I'm with Rackspace for several reasons... two of them are that they've had no power outages and no network outages. They have properly grounded facilities as well as battery and generator backups.

  17. #67

    Lightning Strike - update

    Hi All,

    I just wanted to answer some questions and clarify what happened:

    1. SLA credit - if you'd like one, please log into you PIMS and submit a ticket requesting this. We'll give credit for 10 hours even though the vast majority of servers were down for around 25-35 minutes.

    2. Lightning protection - we do in fact have lightning protection at both facilities in the form of "lightning arrestors". In this case the strike was so severe that the arrestor exploded, but according the the power comany's crew, this is not uncommon. We asked if there was a better lightning protection system, but according to our power company lightning will pretty much destroy anything when the strike is that severe.

    Our UPS also has built in protection, and this appears to have worked exactly as advertised - it immediately sent a signal to all of our battery cabinets and power distribution units telling them to trip their breakers in order to protect all of the equipment. Absolutely no equipment inside the DC was damaged.

    3. What happened?
    The lightning directly hit the pole that we draw power from. It destroyed the cable on the pole and the lightning arrestor first. It then traveled through underground cables, blew the connectors off of one transformer, and blew a hole through part of the underground cable insulation, which caused a severe ground fault. We actually have feeds coming from two different sides of the building for redundancy, but unfortuately there was damage on both sides of the feeds.

    We ran successfully on generator power for most of the day while the power company fixed the cables and transformers. At around 2:30PM we were able to switch back to city power and return to normal electrical operation.

    We of course would like to offer our sincere apologies to anyone that this severely effected. Contrary to some speculation in this thread, we really did put a lot of time, planning and money into building our electrical system. For the most part it did exactly what we would have wanted in a situation as rare and extreme as this. No inside DC equipment was damaged, and the UPS and generators were able to easily keep everthing up as our power company did all of their outside work.

    I hope that this sheds some light on the situation. Thanks for your time.

    Jeremy Hill
    Liquid Web

  18. #68
    Join Date
    Apr 2006
    Location
    italy
    Posts
    0
    Quote Originally Posted by LiquidWebJer
    Hi All,

    I just wanted to answer some questions and clarify what happened:

    1. SLA credit - if you'd like one, please log into you PIMS and submit a ticket requesting this. We'll give credit for 10 hours even though the vast majority of servers were down for around 25-35 minutes.

    2. Lightning protection - we do in fact have lightning protection at both facilities in the form of "lightning arrestors". In this case the strike was so severe that the arrestor exploded, but according the the power comany's crew, this is not uncommon. We asked if there was a better lightning protection system, but according to our power company lightning will pretty much destroy anything when the strike is that severe.

    Our UPS also has built in protection, and this appears to have worked exactly as advertised - it immediately sent a signal to all of our battery cabinets and power distribution units telling them to trip their breakers in order to protect all of the equipment. Absolutely no equipment inside the DC was damaged.

    3. What happened?
    The lightning directly hit the pole that we draw power from. It destroyed the cable on the pole and the lightning arrestor first. It then traveled through underground cables, blew the connectors off of one transformer, and blew a hole through part of the underground cable insulation, which caused a severe ground fault. We actually have feeds coming from two different sides of the building for redundancy, but unfortuately there was damage on both sides of the feeds.

    We ran successfully on generator power for most of the day while the power company fixed the cables and transformers. At around 2:30PM we were able to switch back to city power and return to normal electrical operation.

    We of course would like to offer our sincere apologies to anyone that this severely effected. Contrary to some speculation in this thread, we really did put a lot of time, planning and money into building our electrical system. For the most part it did exactly what we would have wanted in a situation as rare and extreme as this. No inside DC equipment was damaged, and the UPS and generators were able to easily keep everthing up as our power company did all of their outside work.

    I hope that this sheds some light on the situation. Thanks for your time.

    Jeremy Hill
    Liquid Web
    extremely clear

    Thanks jeremy, no problem.

    Now i know what is SLA, and i confirm: we don wanna speculate. only info for our clients.

    LW rox.

    i know rackspace is actually best server solution, but we need a VPS, and rackspace offers only dedicated solutions.

    best regards

    mattep

  19. #69
    Join Date
    Nov 2001
    Posts
    5,383
    Quote Originally Posted by LiquidWebJer
    Hi All,

    I just wanted to answer some questions and clarify what happened:

    1. SLA credit - if you'd like one, please log into you PIMS and submit a ticket requesting this. We'll give credit for 10 hours even though the vast majority of servers were down for around 25-35 minutes.
    I just want to say, I'm not your customer but that's bloody outstanding. Many don't even offer credits when they are down an hour or longer. That's superb

  20. #70
    Join Date
    Jun 2006
    Location
    Earth
    Posts
    98
    Yeah I wish I didn't have to close my old server with LiquidWeb. They offered some of the best services money can buy.

    I do look forward to renting a server from them again when I have the budget and need for it.

    Regards,
    Drac

  21. #71
    Join Date
    Jul 2003
    Posts
    469
    I am not a customer but having deep knowledge of electrical systems I cannot fault LW's response or breif downtime. When a strike hits the main power line, losing power is sometimes unavoidable, regardless of protection in/on the building. I'm actually very suprised the UPS's weren't destroyed, at the very least. Opening a fuse isn't going to stop the energy burst of lightening from jumping across.... sooo if the story is truely as told, wow, consider your data at LW lucky to be alive!!

  22. #72
    I wonder how many web hosts use the center and were affected yesterday.

  23. #73
    Well done Liquidweb.

Page 3 of 3 FirstFirst 123

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •