Page 1 of 7 1234 ... LastLast
Results 1 to 40 of 268
  1. #1
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44

    Exclamation Hurricane Sandy - Outages NY/NJ/CT

    75 Broad Street - Going Down
    The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost.

  2. Thread Summary The following locations were/are experiencing issues due to Hurricane Sandy.

    • 75 Broad Street
    • 33 Whitehall
    • 111 8th Avenue
    • 882 3rd Ave
    • 121 Varick


    (The WHIR) Noise Filter: Hurricane Sandy Floods NYC Data Center, Impacts Hosts, Colocation Providers

    Data Center Knowledge: In Sandy's Aftermath, Epic Challenges for Data Centers

    Contributors: Orien

  3. #2
    Join Date
    Mar 2010
    Posts
    1,152
    Geez, really astonishing what this storm is doing to those cities. How is flooding to the building itself?

  4. #3
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    33 Whitehall - Cogent is Down

  5. #4
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    nLayer link is down at 882 3rd Ave.

  6. #5
    Voxel/Internap are down at 111 8th Avenue

  7. #6
    Join Date
    Mar 2004
    Location
    New Jersey
    Posts
    793
    Quote Originally Posted by voip_expert View Post
    nLayer link is down at 882 3rd Ave.
    ezzi.net is also in that facility, I believe. Anyone heard their situation there? I saw some boxes there come back a short time ago, then they dropped off again after about 20-30 minutes.

  8. #7
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    Quote Originally Posted by JonBiloh View Post
    Voxel/Internap are down at 111 8th Avenue
    Our uplink still up with them and performing very good!

  9. #8
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    Quote Originally Posted by serversphere View Post
    ezzi.net is also in that facility, I believe. Anyone heard their situation there? I saw some boxes there come back a short time ago, then they dropped off again after about 20-30 minutes.
    Facility it self is up, and all operational so far XO and nLayer only have they circuits down there, Cogent, Verizon are just fine.

  10. #9
    Quote Originally Posted by JonBiloh View Post
    Voxel/Internap are down at 111 8th Avenue
    Is this the LGA6 facility?

  11. #10
    Quote Originally Posted by R1CH View Post
    Is this the LGA6 facility?
    Yes, LGA6. I believe it is fed by LGA11.

  12. #11
    Join Date
    Jul 2001
    Location
    Somewhere you're not
    Posts
    229
    I can confirm my server with Voxel/Internap is down at LGA6 is offline. That's at 111 8th Avenue.

  13. #12
    Ah, guess that explains it. Got the warning emails about LGA11, but our box is at LGA6.

  14. #13
    Join Date
    Jul 2001
    Location
    Somewhere you're not
    Posts
    229
    I did too and was thinking "wow, glad I'm not in LGA11!" And then...mine went down! Good to see I'm not the only one, thanks for posting here guys.

    Hopefully we hear from Internap soon. I'm not going to bother them because I'm sure they know there's a problem!

  15. #14
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    Voxel in 7 Teleport operational just fine.

  16. #15
    Join Date
    Jul 2011
    Posts
    496
    Here is more insight information about Internap NY:

    InterNAP is reporting that utility power is no longer available to our NYC location and it has been successfully transferred to generators, keeping all services online.
    InterNAP should have enough on-site fuel for multiple days, and refueling contracts to provide more as necessary.
    InterNAP reports that the facility itself has 60+ hours of fuel but that the basement is now flooded, preventing it from being pumped up to the generators. The site can sustain power for another 4-6 hours on the fuel currently available, they say. It is possible that waters will recede by then. We have posted an event to all NYC customers with all the details available.
    o real whoops so far -- InterNAP seems to be handling things as well as they can and I haven't seen them making mistakes with this in the last 24 hours. I just hope that the flood waters abate quickly in order to minimize downtime. Power outages aren't the end of the world as long as there's minimal corruption -- mostly a nuisance.

    This is from NFoServers FB page.

  17. #16
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    Cogent have gone down at 12:03:42 AM at 882 3rd Ave.

  18. #17
    Join Date
    Sep 2011
    Location
    New York
    Posts
    188
    Quote Originally Posted by JonBiloh View Post
    Voxel/Internap are down at 111 8th Avenue
    Good thing I went with my home town vs NYC
    Last edited by VSNX-Nick; 10-30-2012 at 12:20 AM.

  19. #18
    Join Date
    Feb 2006
    Location
    New York, NY
    Posts
    704
    We're in InterNAP NYM008 (111 8th - 10th Floor), all our gear is still up at this time.

  20. #19
    Join Date
    Sep 2010
    Location
    Philadelphia
    Posts
    406
    The Voxel issues are being caused by a power issue in XO's suite in 111 8th.

  21. #20
    Join Date
    Feb 2002
    Location
    New York, NY
    Posts
    4,612
    Quote Originally Posted by voip_expert View Post
    Our uplink still up with them and performing very good!
    Our Voxel link has been up the whole time as well. We go through their LGA3 gear at 111 8th.

  22. #21
    Join Date
    Dec 2001
    Location
    127.0.0.1
    Posts
    3,640
    Datagram is down too (33 Whitehall - not Cogent)

  23. #22
    Join Date
    Sep 2010
    Location
    Philadelphia
    Posts
    406
    121 Varick has lost power - generator exhaust backing up.

  24. #23
    Join Date
    May 2008
    Location
    Texas
    Posts
    188
    LGA4 (121 Varick St) is done with no ETA

    https://support.steadfast.net/News/N...6-power-outage

  25. #24

    Voxel sites

    Could we get a location code for Voxel sites tacked into the top of this thread where the addresses are showing? And is anyone else unable to access their ubersmith control panels right now?

  26. #25

    Voxel Call

    I got a hold of someone at Internap/Voxel. They said there was an electrical panel issue at the LGA8/LGA9 facility. It should be up within hours. LGA11 does not have a good forecast at this time. Ubersmith is in LGA8/LGA9 and they are trying to relocate it.

    Edit: Voxel/Internap at 111 8th Avenue is LGA8/LGA9

  27. #26
    Join Date
    Jan 2001
    Location
    Miami, FL
    Posts
    1,072

  28. #27
    Join Date
    Aug 2003
    Location
    127.0.0.1
    Posts
    273
    The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.
    http://pastebin.com/9Fr2eW6U - pretty chilling stuff.

  29. #28
    Join Date
    Apr 2012
    Location
    Niceville, Florida
    Posts
    192
    Ouch... Some of my stuff is not far away, but not in the city itself, and all I've dealt with so far is some chucklehead deciding that NOW would be a great time to do "Scheduled Maintenance" on networking gear and not of course let me know that it was scheduled. I honestly am surprised that there hasn't been connectivity issues yet... I'm staying awake for a few more hours to see what happens.

  30. #29
    Join Date
    Nov 2005
    Location
    Sunnyvale, CA
    Posts
    66
    Here's the latest update from Steadfast on their nyc16 facility at 121 Varick St:

    We have just been informed that due to an auxiliary electrical failure, power systems have failed at our nyc16 facility at 121 Varick St in New York City. All New York servers and services are now offline and we have no ETA for service restoration. We will provide updates as we have them.
    Most systems should be configured to power on when power service resumes and we can assist via ticket as needed when power becomes available. We will, of course, honor full SLA credits for the period(s) in which power has been unavailable in the facility. We apologize for any inconvenience this has caused you or your business.
    The following information was provided by our facilities vendor at 121 Varick St, known upstream as LGA4:
    "The auxiliary power for our LGA4 Network Operations Center has gone out. LGA4 data center is will be going down. NOC responses will be delayed as other elements need to be contained to ensure the safety of our crew as we endure the effects of Hurricane Sandy. We are in the process of relocating our team to our near by LGA1 data center.
    At this point it is encouraged that customers who have the ability to, remotely shut down any critical equipment or services to mitigate the risk of any interruptions to service. Our staff is evacuating the site and remote hands services will be suspended until the threat passes. Customer access is also restricted until further notice.
    We will report back as new information becomes available about the condition of our Network Operations Center and Internet Data Center at 121 Varick St."


    I guess it will be a late night bringing up downed websites to other servers..

  31. #30
    Join Date
    Jul 2008
    Location
    Atlanta GA
    Posts
    293
    Spoke with Peer 1. They are running their generators off a gravity-fed tank, so they will be good for quite a while. No outages reported.

  32. #31
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    So far I can say from all our NYC Metro facilities we have been running solid from 7 Teleport Dr, sure we lost multiple carriers and connectivity to few peering in NYC, but other then that we are fully operational.

  33. #32
    Join Date
    Mar 2004
    Location
    New Jersey
    Posts
    793
    ezzi.net looks to be back. Scary stuff coming out of some facilities up there. Thoughts are with you folks...

  34. #33
    Join Date
    Mar 2004
    Location
    New Jersey
    Posts
    793
    Hm, did nLayer just go down again at 882 3rd Ave? All my boxes suddenly dropped off again. They must still be battling...

  35. #34
    Join Date
    Oct 2007
    Location
    Made In NYC
    Posts
    75
    We have racks of servers at 111 Eighth as well as Telehouse NYC. So far TH running beautifully and receiving updates constantly. I can assure you that most of these outages have nothing to do with flooding or the hurricane, and are a direct result of the city's onerous policy to terminate power everywhere below 38th street where every data center is located. Even NYU Hospital is down and relocating their patients because they ran out of generator power. I have been in Manhattan all day long and there is barely any flooding except for all the way downtown, some underground areas and the coastline in very limited areas. And the flooding that does exist in most places downtown is ankle-high. This hurricane has *not* been a big deal for most of NYC. Shutting down power to half of the city was unnecessary and completely overboard. But that is not relevant, data centers should stay online that is why they have generators and proper planning for disasters. I can assure you all most of the power failures reported at these data centers (minus 75 Broad) are related to improper planning, testing, faulty infrastructure, and human error with hosting providers left to throw up their arms in frustration. This is not the first time 111 or XO has screwed up on power. The last time some idiots reversed polarity on the roof.

  36. #35
    Join Date
    Sep 2012
    Location
    NYC
    Posts
    44
    Quote Originally Posted by serversphere View Post
    Hm, did nLayer just go down again at 882 3rd Ave? All my boxes suddenly dropped off again. They must still be battling...
    Evrething is down there since 02:08 AM, I think generators did not kick in or so...

  37. #36
    Join Date
    Apr 2003
    Location
    Los Angeles, CA
    Posts
    244
    Zopim, the live chat software is also down. They seem to be hosted by the affected Internap DC. We use Zopim and have temporarily removed their chat scripts from our website as it was slowing down page requests across our entire site.

    The following well known sites are down: Gawker is down, Gizmodo is down, HuffPost was down (now back up). All hosted by Datagram. Hard to believe these companies don't have any type of geographic redundancy in place. Not even a splash page.

  38. #37
    Join Date
    Feb 2004
    Location
    Louisville, Kentucky
    Posts
    1,083
    Quote Originally Posted by mr fixit View Post
    I can assure you that most of these outages have nothing to do with flooding or the hurricane, and are a direct result of the city's onerous policy to terminate power everywhere below 38th street where every data center is located. Even NYU Hospital
    Almost all of the utility power failures in Manhattan resulted from the substation explosion near FDR Drive.

    The intentional outages were very limited and did not include one single co-location facility mentioned in this thread with the possible exception of 25 Broadway -- I am not totally clear on that one as it is adjacent to the planned outage zone.

    conEdison has not given out any firm repair estimates, of course; but they do not seem very optimistic about how quickly the substation can be repaired, saying only, "up to a week but probably less" for the 1/4-million customers affected by that throughout Manhattan. The Washington Post and CNN have interviewed their spokespeople within the last couple of hours.

  39. #38
    Internap said in there update email that there is power backup fuel for next 6 hours and users can gracefully shutdown there servers but my website is down before the email updated reached me ???

  40. #39
    Quote Originally Posted by SEEKER2007 View Post
    Internap said in there update email that there is power backup fuel for next 6 hours and users can gracefully shutdown there servers but my website is down before the email updated reached me ???
    Internap have multiple affected facilities. LGA11 has the small amount of reserve power which by now almost exhausted. LGA6 dropped off suddenly with what I'm assuming is a power failure, LGA8 is also experiencing a "catastrophic power failure". No official word yet on LGA6.

  41. #40
    Join Date
    Oct 2007
    Location
    Made In NYC
    Posts
    75
    Quote Originally Posted by jsw6 View Post
    Almost all of the utility power failures in Manhattan resulted from the substation explosion near FDR Drive.

    The intentional outages were very limited and did not include one single co-location facility mentioned in this thread with the possible exception of 25 Broadway -- I am not totally clear on that one as it is adjacent to the planned outage zone.
    While the transformer explosion affected half of the city many the providers mentioned on this board announced planned cutover to generator power specifically due to the intentional ConEd shut down in their trouble tickets. Cutovers and outages occurred outside the time of the explosion. For instance XO went down at Midnight, that has nothing to do with ConEd. Peer1 is at 75 Broad in the zone, as is 33 Whitehall. Internap along with many others are at XO, which has power problems. Some of the data centers acknowledged phase drops , operator errors, UPS failures, and power panel problems which are happening in specific suites in these buildings, while the rest of the suites in many of those buildings are running on building generator without any issue. The majority of these data centers really should not be down right now.
    Last edited by mr fixit; 10-30-2012 at 04:24 AM.

Page 1 of 7 1234 ... LastLast

Similar Threads

  1. Hurricane Sandy
    By ViperNet in forum Web Hosting Lounge
    Replies: 76
    Last Post: 11-05-2012, 01:53 PM
  2. Hurricane Irene becomes a MONSTER of a Hurricane
    By KMyers in forum Web Hosting Lounge
    Replies: 51
    Last Post: 08-30-2011, 07:37 PM
  3. monitoring outages
    By coolhand in forum Hosting Security and Technology
    Replies: 11
    Last Post: 03-04-2003, 06:29 AM
  4. Outages
    By WorldNet in forum Dedicated Server
    Replies: 4
    Last Post: 01-15-2002, 04:31 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •