Page 2 of 2 FirstFirst 12
Results 26 to 44 of 44
  1. #26
    I can't phantom an the idea that Inergen could produce damage to electronic equipment. Why would anyone put it into a data center if it does? There are companies and organizations whose loss of data and equipment would result in worldwide catastrophic impact in millions of people's lives.

    Wikipedia article is altered on Feb.20.2010
    http://en.wikipedia.org/w/index.php?...ldid=345309674
    by someone from uk2, the owners of Westhost. That page was altered to say" In some cases the release of Inergen is known to cause damage to hard drives". It was altered from IP 83.170.70.50 - gateway.uk2.net

    http://www.youtube.com/watch?v=BEFQI1DFpYA
    Ansul Inergen Clean Agent System - described as a safe for sensitive computer equipment.
    ===========
    Equipment safe: Unlike many other agents that can become corrosive and permanently damage electronic components, Inergen® does not form any corrosive by-products that can damage equipment in server rooms, data centres, medical facilities, laboratories or other sensitive electronic environments.

    http://www.wormald.com.au/fire-syste...gaseous-system
    ===================

  2. #27
    Join Date
    Feb 2004
    Posts
    48
    Good explanation eming, and I can see how pressure could have catastrophic effects on the hard disks.

    What I'm struggling with is how come you used Inergen (or were satisfied with the use of Inergen) when you had this knowledge about how hard disks react to huge pressure changes in the environment.

  3. #28
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    Yes, we added that - I feel we had the proof to do so, and it was only fair to warn others. Can you blame us?

    SO, it is a coincidence that every colo client in the DC lost drives at the same second?

    Don't get me wrong, I don't actually think it was the only gas itself that killed the drives, my theory is that the DC's ventilation valves were not adjusted probably to release the pressure after killing fire, but before killing drives.
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  4. #29
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    Quote Originally Posted by tryaxiom View Post
    Good explanation eming, and I can see how pressure could have catastrophic effects on the hard disks.

    What I'm struggling with is how come you used Inergen (or were satisfied with the use of Inergen) when you had this knowledge about how hard disks react to huge pressure changes in the environment.
    It is not our DC. And the DC itself is as good as it comes. As expensive as any and fully SAS70-II certified etc.
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  5. #30
    1) "If the air pressure is too low" - pressure wasn't too low, if anything it could had been a bit higher of about 10%, but most likely not more, otherwise people in the facility would had been bleeding from their ears due to such a high pressure increases since Inergen is released as gas to dilute oxygen from 25% to 15%. Also we don't know how much Inergen was released, was all gas was released or only 1 out of 10.

    2) Hard disk drives are not airtight. They have a permeable filter (a breather filter) between the top cover and inside of the drive, to allow the pressure inside and outside the drive to equalize while keeping out dust and dirt. .... You can see these breather holes on all drives -- they usually have a warning sticker next to them, informing the user not to cover the holes.

  6. #31
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    Quote Originally Posted by AlphaOne View Post
    1) "If the air pressure is too low" - pressure wasn't too low, if anything it could had been a bit higher of about 10%, but most likely not more, otherwise people in the facility would had been bleeding from their ears due to such a high pressure increases since Inergen is released as gas to dilute oxygen from 25% to 15%. Also we don't know how much Inergen was released, was all gas was released or only 1 out of 10.

    2) Hard disk drives are not airtight. They have a permeable filter (a breather filter) between the top cover and inside of the drive, to allow the pressure inside and outside the drive to equalize while keeping out dust and dirt. .... You can see these breather holes on all drives -- they usually have a warning sticker next to them, informing the user not to cover the holes.
    Details served to us are still very scarce, so we do not know as well. But I do feel I have all the proof I need to verify how our (and any other client in the DC) drives died.

    Please note, that I have no personal gripe or opinion leaning towards any fire suppression technology, I can only tell you what happened, and that we are still in the middle of a cleanup likely to cost us $500k-1m in SLA credits, lost clients, new equipment and manhours.

    I will be happy to update this thread when I get more details.


    D
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  7. #32
    If this was a pressure issue, which is doubtful because same parts are usually being used in all facilities for fire suppression and they are presumably fool proof, then its facility owner is to blame or their vendor who installed system that can catastrophically impact environment. And they should pay from their insurance to all of us.

  8. #33
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    Quote Originally Posted by AlphaOne View Post
    And they should pay from their insurance to all of us.
    I obviously agree
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  9. #34
    Please do update on cause of hard drive failure.

    Also, Westhost told me that not only hard drives had failed, but also other computer components had failed (fried) and that they had to create servers from scratch.

    What could had caused motherboard or CPU to fry ?
    They all come with thermal protection nowadays.

  10. #35
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    It is true that we've created a whole lot of servers from scratch while working on rescuing data from the original servers.
    I have not heard of any other equipment failing that hard drives. I know that the few hundred servers powering VPS.NET in SLC *only* had hard drive errors, and the servers running SSD's had no problems at all.

    Also, I hope you can forgive some of the level1 ticket/chat/phone techs for not having full info, please appreciate that the level3 techs literally worked for +40 hours to get over the initial hurdle, and they may not have been able to fully inform all levels of the organization. Or the Level1 techs might not have been able to fully comprehend the technical implications of what happened - I know I did not for a while.

    This is by far the worst situation I've seen at a DC for the 15 years I've been in the industry. It is far more damaging than power/network/etc issues I've been through in the past.

    I will make sure to update this thread when I have more info.


    D
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  11. #36
    Join Date
    Oct 2003
    Posts
    9,264
    If I'm not mistaken (at least from what I've read, admittedly I have zero first-hand experience with any kind of fire suppressant systems) -- wouldn't the pressure in the facility need to be relieved in order for it to avoid damage? And in this particular case, was that not done, causing all of this damage?

    e.g. the build-up of pressure, not the fire suppressant system itself.
    According to wikipedia listed under the disadvantages of the inergen system:

    "Inergen requires that 40-50% of the room atmosphere be replaced with Inergen in a short amount of time. This creates a large amount of pressure, which must be relieved to prevent damage to the enclosure."

  12. #37
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    Quote Originally Posted by David View Post
    If I'm not mistaken (at least from what I've read, admittedly I have zero first-hand experience with any kind of fire suppressant systems) -- wouldn't the pressure in the facility need to be relieved in order for it to avoid damage? And in this particular case, was that not done, causing all of this damage?

    e.g. the build-up of pressure, not the fire suppressant system itself.
    According to wikipedia listed under the disadvantages of the inergen system:

    "Inergen requires that 40-50% of the room atmosphere be replaced with Inergen in a short amount of time. This creates a large amount of pressure, which must be relieved to prevent damage to the enclosure."
    Quote Originally Posted by eming View Post
    Don't get me wrong, I don't actually think it was the only gas itself that killed the drives, my theory is that the DC's ventilation valves were not adjusted probably to release the pressure after killing fire, but before killing drives.
    Yup - my theory as well. That is just a guess from my side though.
    Well, it *was* the fire suppressant that killed the drives, as in, without it our drives would be fine now.

    By the way, I should highlight that Interxion has seen the exact same situation in their DC in Denmark. http://www.version2.dk/artikel/11485...mpaign=nyheder (use Google translate to get the gist of it)
    Fire suppressant also killed a large number of drives there.


    D
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  13. #38
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    btw, check out this vid, showing how sensitive disks are: http://www.youtube.com/watch?v=tDacjrSCeq4
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  14. #39
    Ok, www.version2.dk article is interesting, but google translated it as
    He stresses that any errors in hardware is not due to pressure from the gas has pushed to the device or something similar.
    Original:
    Han understreger, at eventuelle fejl på hardware ikke skyldes, at trykket fra gassen har skubbet til udstyret eller noget lignende.

    Is there a Danish to English expert ?

    As for shouting at hard drive resulting in slowdowns, it wasn't scientific test. Just some anomaly that could had caused by him touching the drive.
    Last edited by AlphaOne; 02-23-2010 at 09:40 AM.

  15. #40
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    Quote Originally Posted by AlphaOne View Post
    Ok, www.version2.dk article is interesting, but google translated it as


    Original:
    Han understreger, at eventuelle fejl på hardware ikke skyldes, at trykket fra gassen har skubbet til udstyret eller noget lignende.

    Is there a Danish to English expert ?
    I am Danish
    Our experience was the same. All the gear was still probably racked up, from the outside it looked pretty normal, and there was no physical damage to be seen anywhere.

    Also, the article does not reveal if the damaged caused by the fire suppressant was as serious was in our case, but it very clearly state that it did cause damage to the hard drives and storage units.

    Quote Originally Posted by AlphaOne View Post
    As for shouting at hard drive resulting in slowdowns, it wasn't scientific test. Just some anomaly that could had caused by him touching the drive.
    admitted, that post was a bit offtopic, but still interesting.


    D
    Last edited by eming; 02-23-2010 at 09:45 AM.
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  16. #41
    So is the translation correct?
    "He stresses that any errors in hardware is not due to pressure from the gas has pushed to the device or something similar. "

    Sound could be somewhat related if it was on resonating frequency.

  17. #42
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,409
    yes, it is correct - it is "not due to pressure from the gas has pushed to the device" - thats the whole point, no servers were pushed around, fallen to the floor, bumped up or blown to pieces. As far as I know it *only* caused damage to the drives inside the servers because of what Matt explained earlier.

    It seems you have an agenda here honestly
    I can only tell you about our experiences, and confirm that other DC (Interxion in this case) has had a similar experience.
    I have no agenda, only a genuine interest in finding out what the root cause of the faulty drives are. So far it looks like a no-brainer to me. (1) Drives were fine until gas was released. (2) It was not a UK2Group specific issue as all the other colo clients in the same DC also had faulty drives and (3) other reputable DC's (InterXion) has gone through what seems to be the exact same situation.

    I will get back to work now, and leave this thread alone until I have more info, by then I will update as promised (IF I get more info, that is).


    D
    Ditlev Bredahl. CEO,
    OnApp.com + Cloud.net & CDN.net

  18. #43
    It seems you have an agenda here honestly
    No I don't have sinister agenda, except I need to find out what went wrong so I can choose a DC that wouldn't have that kind of issue, since a lot of systems have Inergen nowadays.

    As for "..no servers were pushed around, fallen to the floor, bumped up or blown to pieces..." the way its translated to english it sounds like not wind blowing things from their place, but that its "not an atmospheric pressure reaction" or maybe translation is not quite accurate and indeed it meant to say that its "not due to a shock wave".

    Any info how fire-suppression system could had caused the problem would be appreciated.

  19. #44
    Ditlev, hundreds of frantic Westhost users are on your own Westhost forum begging for any information they can get about what the hell is going on. Mostly, they are not getting any useful replies but instead ponderous "officIal" releases, which often contain information that the reader knows is untrue - for example, list servers as repaired and functioning that are not). This does not reassure them! Others have tried the solution offered by Westhost, to immediately create a new WH4 site for them, only to find themselves stuck in limbo when the "instant" DNS transfer doesn't take place and tech support cannot be reached.

    This being the case, what on earth are you doing here pontificating about fire suppression technology? Plenty of time for that later, when the current mess is cleaned up. If you were the pilot of a plane that lost all power, would you fire up you laptop and go to a Pilots Forum to discuss how to avoid it happening next time, or would you land the damn plane?

    Get over to the Westhost forum and start answering questions. If you don't know the answers, find them. You are CEO, you ought to be able to do a better job of it than that poor manager you got doing it now.

Page 2 of 2 FirstFirst 12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •