Page 1 of 2 12 LastLast
Results 1 to 40 of 44
  1. #1
    Join Date
    Dec 2004
    Location
    Rainy Oregon!
    Posts
    47

    Inergen vs. Water in Server Room?

    I work for a large company that has dedicated server rooms for all of our servers + our PBX.

    At our building we have inergen as the fire retardant for our server room, but I was just talking to another Facility Manager who said she was told (by who???) that inergen causes as much damage to servers as water does, so on this advice she was putting sprinkler systems into all her server rooms in new building...

    Does anyone have any experience with this? Either the water or the inergen?

  2. #2
    Join Date
    Nov 2000
    Location
    USA
    Posts
    1,269
    I'm no expert but I don't think you'd want to use water in an area where there is the potential for electrical fire.

  3. #3
    Join Date
    Nov 2004
    Location
    Dallas
    Posts
    739
    I dont agree with that, just imagine water in a server... men.. a must see situation (but not my servers)

  4. #4
    Join Date
    Dec 2004
    Location
    Rainy Oregon!
    Posts
    47
    Thanks... my thoughts exactly... I can't imagine mixing water & servers!

  5. #5
    Join Date
    Jun 2004
    Posts
    1,958
    You wonder if its possible to make a server room with no oxygen..then fire cant even start..lol

  6. #6
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,515
    Originally posted by camers
    You wonder if its possible to make a server room with no oxygen..then fire cant even start..lol
    Sure, then nobody could work.. Briillliant idea.
    I'm sorry, but our last oxygen mask was just used, and the guy STILL hasn't come out of the server room yet
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Linux Problems? WHMCS Issues? +1-866-546-8914 (linux-14) or @whmcsguru on twitter!

  7. #7
    Join Date
    Jun 2004
    Posts
    1,958
    haha.. Im sure theres a way round it. Im thinking a masisve generator that sucks it all out some how and sorts the whole ion particals out etc etc

    be pretty sweet... we'll see datacenters in 10 years time haha

  8. #8
    Join Date
    Nov 2004
    Location
    Dallas
    Posts
    739
    LOL

    Datacenter :"Sorry we have some problem with our MGTSIAOSH (massive generator that sucks it all out some how). It went crazy and sucked the technician that was going to reboot your server"


    And in the "Datacenter" section of thousands of webhosts, you are going to see in the future: Plasma Generator Power, UPS backups, MGTSIAOSH,.



    I cant stop laughing.... hahahahahahahha

  9. #9
    Join Date
    Dec 2004
    Location
    Rainy Oregon!
    Posts
    47
    I'll check to see if the MGTSIAOSH is in the 2005 budget

    (and the special masks that will have to be implemented to keep everyone's eyes from getting sucked out of their heads while working in there) who's gonna tell my telephony guy???


  10. #10
    Join Date
    Jun 2004
    Posts
    1,958
    heh amusing..thanks guys :p

  11. #11
    Join Date
    Dec 2002
    Location
    The Shadows
    Posts
    2,913
    You want some sort of gas suppression system.

    If you do that though, you most likely need oxygen masks placed at regular intervals on the DC floor incase some of your employees get caught when the system goes off.

    Might be something to look at, anyways...

    But water and electricity == bad
    Dan Sheppard ~ Freelance whatever

  12. #12
    Join Date
    Jan 2002
    Location
    Hudson, Wisconsin
    Posts
    560
    Take a look as something that the company I work for just came out with. It is pretty cool stuff. We submerged a powered up laptop and cell phone in it and it kept working. Here are some links.

    3M Novec 1230 Fire Protection Fluid.

    http://cms.3m.com/cms/US/en/2-68/iclcrFR/view.jhtml

    http://wcco.com/siteSearch/local_story_350105453.html

  13. #13
    Join Date
    Dec 2004
    Location
    Rainy Oregon!
    Posts
    47
    Thanks for the links & info. We literally just found out Wednesday that we had server rooms being designed w/ water sprinklers... and want to make sure we put an end to that - quickly


  14. #14
    Join Date
    Sep 2004
    Location
    Chicago, IL
    Posts
    213
    Ben Lenard, MS, MBA
    TechMinds 4 Hire, Inc - (866) 214-1285 x 2001
    http://www.tm4h.com

  15. #15
    Join Date
    Dec 2002
    Location
    California
    Posts
    2,005
    Where some of our servers are colocated in a Time Warner building, it uses FM-200.

    http://www.reliablefire.com/fm200/fm200.html
    http://www.e1.greatlakes.com/fm200/jsp/index.jsp

    It's not deadly to humans, but it certainly isn't good. Then again you probably shouldn't be in the datacenter when it's on fire.
    I wish all my traffic went through AS174.

  16. #16
    Join Date
    Jun 2002
    Location
    Sydney, Australia
    Posts
    1,745
    Originally posted by blenard
    Did you look into Halon?
    I didn't think Halon was allowed to be used anymore? I'm not even sure if new suppressant is being manufactured, as Halon is a CFC.

    As for fire suppression, FM-200 seems to be a good option from what I've read about it, and it doesn't require people to go racing for oxygen masks if the system is discharged.

    It's probably a good idea to consider a pre-action sprinkler as a last resort if something like FM-200 doesn't work, especially if you can setup a system to cut power to all or part of a server room in case of a fire - especially if the server room is in an office building (building code would probably require sprinklers in this case anyway).

    -Shaun

  17. #17
    FM-200 is the way to go. If setup properly, FM-200 is the last resort, no need to take any actions after it goes off because it will displace all oxygen in the room. No oxygen, no fire. Keep servers as far away from water as possible...

    T

  18. #18
    Join Date
    Sep 2004
    Location
    Chicago, IL
    Posts
    213
    I know someone that still sells Halon sysztems. not sure..
    Ben Lenard, MS, MBA
    TechMinds 4 Hire, Inc - (866) 214-1285 x 2001
    http://www.tm4h.com

  19. #19
    Searching the net I can't find any reputable report of Inergen damaging the computer equipment or data. If you find it please let me know.

  20. #20
    search again,
    find the vendor,
    read the application notes.

    westhost, right?
    edgedirector.com
    managed dns global failover and load balance (gslb)
    exactstate.com
    uptime report for webhostingtalk.com

  21. #21
    Join Date
    Feb 2004
    Posts
    43
    plumsauce, we have no evidence that Inergen caused the damage here. In fact, Westhost have been obfuscating on the issue ever since the original problem occured several days ago.

    Any host that goes down for a whole week because the fire protection system was triggered needs to point a finger somewhere. For the moment WH is trying to imply that it's the Inergen that's the cause of all their servers getting damaged ...while at the same time not categorically saying that.

  22. #22
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Quote Originally Posted by tryaxiom View Post
    plumsauce, we have no evidence that Inergen caused the damage here. In fact, Westhost have been obfuscating on the issue ever since the original problem occured several days ago.

    Any host that goes down for a whole week because the fire protection system was triggered needs to point a finger somewhere. For the moment WH is trying to imply that it's the Inergen that's the cause of all their servers getting damaged ...while at the same time not categorically saying that.
    I can confirm 100% that it was the Inergen based fire protection system that caused all the issues we had in the Salt Lake City DC.

    http://status.vps.net/?p=136 <- read more there. (VPS.NET shares DC space with Westhost - both owned by UK2Group.com)


    D
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  23. #23
    I am not sure if you can actually confirm that Inergen discharge caused hardware failure, and just saying that you can is not a proof of confirming it.

    Report from Consonus http://status.vps.net/wp-content/upl...010/02/cip.pdf is of Inergen discharge, not that it caused such a widespread computer hardware failure.

    Do you have any proof ?

  24. #24
    Join Date
    Feb 2004
    Posts
    43
    Thanks for your reply.

    the Inergen based fire protection system that caused all the issues
    That's a significant departure from anything that's been said on the matter so far.

  25. #25
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Quote Originally Posted by AlphaOne View Post
    I am not sure if you can actually confirm that Inergen discharge caused hardware failure, and just saying that you can is not a proof of confirming it.

    Report from Consonus http://status.vps.net/wp-content/upl...010/02/cip.pdf is of Inergen discharge, not that it caused such a widespread computer hardware failure.

    Do you have any proof ?
    Proof? Well, all the proof I need.

    At 2PM all systems etc were fine
    At 2.21PM the Inergen suppression system was released (human error it seems)
    10 mins later or so, we lost ~300 hard drives

    And it was not only us, all the other COLO clients in the DC were running around replacing drives like mad men.

    Asking my brilliant IT manager I got this answer:

    (...)Essentially, due to an accidental release of Inergen during a routing
    fire inspection yesterday the sudden increase of air pressure on the
    data center floor resulted in mass disk failures, files corruption, and
    failed arrays. See below for a more detailed explanation:

    The (mostly) sealed enclosure protects the drive internals from dust,
    condensation, and other sources of contamination. The hard disk's
    read-write heads fly on an air bearing (a cushion of air) only
    nanometers above the disk surface. The disk surface and the drive's
    internal environment must therefore be kept immaculately clean, as
    fingerprints, hair, dust, and even smoke particles have mountain-sized
    dimensions when compared to the submicroscopic gap that the heads
    maintain.

    Some people believe a disk drive contains a vacuum - this is incorrect,
    as the system relies on air pressure inside the drive to support the
    heads at their proper flying height while the disk is in motion. Another
    common misconception is that a hard drive is totally sealed. A hard disk
    drive requires a certain range of air pressures in order to operate
    properly. If the air pressure is too low, the air will not exert enough
    force on the flying head, the head will not be at the proper height, and
    there is a risk of head crashes and data loss. (Specially manufactured
    sealed and pressurized drives are needed for reliable high-altitude
    operation, above about 10,000 feet.) Some modern drives include flying
    height sensors to detect if the pressure is too low, and temperature
    sensors to alert the system to overheating problems.

    Best regards,

    Matt McBride (CCNP, CCDP, CISSP)
    IT Manager

    D
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  26. #26
    I can't phantom an the idea that Inergen could produce damage to electronic equipment. Why would anyone put it into a data center if it does? There are companies and organizations whose loss of data and equipment would result in worldwide catastrophic impact in millions of people's lives.

    Wikipedia article is altered on Feb.20.2010
    http://en.wikipedia.org/w/index.php?...ldid=345309674
    by someone from uk2, the owners of Westhost. That page was altered to say" In some cases the release of Inergen is known to cause damage to hard drives". It was altered from IP 83.170.70.50 - gateway.uk2.net

    http://www.youtube.com/watch?v=BEFQI1DFpYA
    Ansul Inergen Clean Agent System - described as a safe for sensitive computer equipment.
    ===========
    Equipment safe: Unlike many other agents that can become corrosive and permanently damage electronic components, Inergen® does not form any corrosive by-products that can damage equipment in server rooms, data centres, medical facilities, laboratories or other sensitive electronic environments.

    http://www.wormald.com.au/fire-syste...gaseous-system
    ===================

  27. #27
    Join Date
    Feb 2004
    Posts
    43
    Good explanation eming, and I can see how pressure could have catastrophic effects on the hard disks.

    What I'm struggling with is how come you used Inergen (or were satisfied with the use of Inergen) when you had this knowledge about how hard disks react to huge pressure changes in the environment.

  28. #28
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Yes, we added that - I feel we had the proof to do so, and it was only fair to warn others. Can you blame us?

    SO, it is a coincidence that every colo client in the DC lost drives at the same second?

    Don't get me wrong, I don't actually think it was the only gas itself that killed the drives, my theory is that the DC's ventilation valves were not adjusted probably to release the pressure after killing fire, but before killing drives.
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  29. #29
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Quote Originally Posted by tryaxiom View Post
    Good explanation eming, and I can see how pressure could have catastrophic effects on the hard disks.

    What I'm struggling with is how come you used Inergen (or were satisfied with the use of Inergen) when you had this knowledge about how hard disks react to huge pressure changes in the environment.
    It is not our DC. And the DC itself is as good as it comes. As expensive as any and fully SAS70-II certified etc.
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  30. #30
    1) "If the air pressure is too low" - pressure wasn't too low, if anything it could had been a bit higher of about 10%, but most likely not more, otherwise people in the facility would had been bleeding from their ears due to such a high pressure increases since Inergen is released as gas to dilute oxygen from 25% to 15%. Also we don't know how much Inergen was released, was all gas was released or only 1 out of 10.

    2) Hard disk drives are not airtight. They have a permeable filter (a breather filter) between the top cover and inside of the drive, to allow the pressure inside and outside the drive to equalize while keeping out dust and dirt. .... You can see these breather holes on all drives -- they usually have a warning sticker next to them, informing the user not to cover the holes.

  31. #31
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Quote Originally Posted by AlphaOne View Post
    1) "If the air pressure is too low" - pressure wasn't too low, if anything it could had been a bit higher of about 10%, but most likely not more, otherwise people in the facility would had been bleeding from their ears due to such a high pressure increases since Inergen is released as gas to dilute oxygen from 25% to 15%. Also we don't know how much Inergen was released, was all gas was released or only 1 out of 10.

    2) Hard disk drives are not airtight. They have a permeable filter (a breather filter) between the top cover and inside of the drive, to allow the pressure inside and outside the drive to equalize while keeping out dust and dirt. .... You can see these breather holes on all drives -- they usually have a warning sticker next to them, informing the user not to cover the holes.
    Details served to us are still very scarce, so we do not know as well. But I do feel I have all the proof I need to verify how our (and any other client in the DC) drives died.

    Please note, that I have no personal gripe or opinion leaning towards any fire suppression technology, I can only tell you what happened, and that we are still in the middle of a cleanup likely to cost us $500k-1m in SLA credits, lost clients, new equipment and manhours.

    I will be happy to update this thread when I get more details.


    D
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  32. #32
    If this was a pressure issue, which is doubtful because same parts are usually being used in all facilities for fire suppression and they are presumably fool proof, then its facility owner is to blame or their vendor who installed system that can catastrophically impact environment. And they should pay from their insurance to all of us.

  33. #33
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Quote Originally Posted by AlphaOne View Post
    And they should pay from their insurance to all of us.
    I obviously agree
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  34. #34
    Please do update on cause of hard drive failure.

    Also, Westhost told me that not only hard drives had failed, but also other computer components had failed (fried) and that they had to create servers from scratch.

    What could had caused motherboard or CPU to fry ?
    They all come with thermal protection nowadays.

  35. #35
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    It is true that we've created a whole lot of servers from scratch while working on rescuing data from the original servers.
    I have not heard of any other equipment failing that hard drives. I know that the few hundred servers powering VPS.NET in SLC *only* had hard drive errors, and the servers running SSD's had no problems at all.

    Also, I hope you can forgive some of the level1 ticket/chat/phone techs for not having full info, please appreciate that the level3 techs literally worked for +40 hours to get over the initial hurdle, and they may not have been able to fully inform all levels of the organization. Or the Level1 techs might not have been able to fully comprehend the technical implications of what happened - I know I did not for a while.

    This is by far the worst situation I've seen at a DC for the 15 years I've been in the industry. It is far more damaging than power/network/etc issues I've been through in the past.

    I will make sure to update this thread when I have more info.


    D
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  36. #36
    Join Date
    Oct 2003
    Location
    Chattanooga
    Posts
    8,976
    If I'm not mistaken (at least from what I've read, admittedly I have zero first-hand experience with any kind of fire suppressant systems) -- wouldn't the pressure in the facility need to be relieved in order for it to avoid damage? And in this particular case, was that not done, causing all of this damage?

    e.g. the build-up of pressure, not the fire suppressant system itself.
    According to wikipedia listed under the disadvantages of the inergen system:

    "Inergen requires that 40-50% of the room atmosphere be replaced with Inergen in a short amount of time. This creates a large amount of pressure, which must be relieved to prevent damage to the enclosure."
    David
    Fused.com — web hosting for businesses that don't want to think about web hosting.
    Follow me on twitter @davidandgoliath

  37. #37
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Quote Originally Posted by David View Post
    If I'm not mistaken (at least from what I've read, admittedly I have zero first-hand experience with any kind of fire suppressant systems) -- wouldn't the pressure in the facility need to be relieved in order for it to avoid damage? And in this particular case, was that not done, causing all of this damage?

    e.g. the build-up of pressure, not the fire suppressant system itself.
    According to wikipedia listed under the disadvantages of the inergen system:

    "Inergen requires that 40-50% of the room atmosphere be replaced with Inergen in a short amount of time. This creates a large amount of pressure, which must be relieved to prevent damage to the enclosure."
    Quote Originally Posted by eming View Post
    Don't get me wrong, I don't actually think it was the only gas itself that killed the drives, my theory is that the DC's ventilation valves were not adjusted probably to release the pressure after killing fire, but before killing drives.
    Yup - my theory as well. That is just a guess from my side though.
    Well, it *was* the fire suppressant that killed the drives, as in, without it our drives would be fine now.

    By the way, I should highlight that Interxion has seen the exact same situation in their DC in Denmark. http://www.version2.dk/artikel/11485...mpaign=nyheder (use Google translate to get the gist of it)
    Fire suppressant also killed a large number of drives there.


    D
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  38. #38
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    btw, check out this vid, showing how sensitive disks are: http://www.youtube.com/watch?v=tDacjrSCeq4
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

  39. #39
    Ok, www.version2.dk article is interesting, but google translated it as
    He stresses that any errors in hardware is not due to pressure from the gas has pushed to the device or something similar.
    Original:
    Han understreger, at eventuelle fejl på hardware ikke skyldes, at trykket fra gassen har skubbet til udstyret eller noget lignende.

    Is there a Danish to English expert ?

    As for shouting at hard drive resulting in slowdowns, it wasn't scientific test. Just some anomaly that could had caused by him touching the drive.
    Last edited by AlphaOne; 02-23-2010 at 09:40 AM.

  40. #40
    Join Date
    Sep 2005
    Location
    London
    Posts
    2,404
    Quote Originally Posted by AlphaOne View Post
    Ok, www.version2.dk article is interesting, but google translated it as


    Original:
    Han understreger, at eventuelle fejl på hardware ikke skyldes, at trykket fra gassen har skubbet til udstyret eller noget lignende.

    Is there a Danish to English expert ?
    I am Danish
    Our experience was the same. All the gear was still probably racked up, from the outside it looked pretty normal, and there was no physical damage to be seen anywhere.

    Also, the article does not reveal if the damaged caused by the fire suppressant was as serious was in our case, but it very clearly state that it did cause damage to the hard drives and storage units.

    Quote Originally Posted by AlphaOne View Post
    As for shouting at hard drive resulting in slowdowns, it wasn't scientific test. Just some anomaly that could had caused by him touching the drive.
    admitted, that post was a bit offtopic, but still interesting.


    D
    Last edited by eming; 02-23-2010 at 09:45 AM.
    Ditlev Bredahl. CEO,
    OnApp.com & SolusVM.com + Cloud.net & CDN.net

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •