Web Hosting Talk : Web Hosting Main Forums : Industry Announcements : Providers and Network Outages and Updates : TELEHOUSE 25 Broadway - POWER

[lostpacket]

Dear valued customer,

We extremely apologize for the unexpected power failure that was experienced to your equipment today. Telehouse has restored power to your equipments at this moment , currently our UPS vendor is investigating the cause of failure. If you still experience outage to any of your equipment please contact CAC at 718-355-2525 we will work with you to restore the power immediately. Also, CAC will be notifying the status update every two hours based on the vendor's diagnosis.

Thank you so much for your cooperation and understanding.

Regards,

Partheepan "Raj" Rajaratnam
TELEHOUSE America
Assistant Manager
Facilities Engineering & Operation

[djahandarie]

Really quite silly. UPS issue which caused a multi-hour outage. *sigh*

[lostpacket]

Not the first time this has happened at 25 Broadway. Its been a few years though.

[bqinternet]

Ah, that brings back memories. I think they had 2 outages when we were there years ago. The second outage didn't affect us, because after the first, we had them run us a redundant circuit from another floor.

[CGotzmann]

Originally Posted by djahandarie

Really quite silly. UPS issue which caused a multi-hour outage. *sigh*

Quite silly? This stuff happens.

[djahandarie]

Originally Posted by CGotzmann

Quite silly? This stuff happens.

They lacked the redundancy, and/or the ability to bypass the UPS, and/or the ability to get the vendor/replacements on scene quickly. It does happen, but that doesn't mean I'm not going to complain about it .

[bqinternet]

Originally Posted by CGotzmann

Quite silly? This stuff happens.

When it happens every 2-3 years, it's a different story. It's my understanding that the outages we experienced there were not their first.

[BurstNET]

Aren't they closing that facility soon?...I think so.
.
.

[freethought]

Yeah, 25 Broadway is due to be shuttered in mid 2013 I think.

[TadasPet]

Here is update from TELEHOUSE as of 05/01/2012

Dear Valued Customer,

The following is a recap of activity and latest update on the power outage that
occurred on Sunday, 4/29 affecting only UPS-A Customers at the TELEHOUSE 25 Broadway
facility, in New York City.

On Sunday evening at around 6PM, there was a commercial utility power drop. At that
time, the primary UPS - A1 system failed to maintain critical power load apparently
due to a fuse failure. The UPS A2 system, which is the failover or backup system to
UPS A1, failed to carry the shifting load for undetermined reasons at this time. The
diesel generators also failed to start again due to undetermined reasons at this
time.
Critical Power was restored using diesel generator at approximately 9:00PM Sunday
and has been stable since that time.

Current Status and Activities Timeline

<4/30>
- Our UPS vendor identified 2 failed (blown) inverter fuses and　replaced all
(6) the fuses on the UPS A1 inverter.
- No failed (blown) fuses were identified in UPS A2.
- UPS vendor swapped out the faulty inverter on UPS A1 and SCR tested new inverter
for proper operation.
- All the batteries on both units were load tested for voltage.

<5/1>
- Commercial utility power feeds are stable at this time.
- Diesel generator power is stable.
- Our UPS vendor performed the check on UPS#A2 this morning, and turned it on now
with no-commercial load for testing.
- Other proactive trouble shooting and testing is underway on UPS A systems.
- All the load for UPS A system will be continue to be supported by generator system
until UPS A is fully restored.
- We are in the stage of finalizing the testing the power stability both for
UPS-A1&2, and the transfer is scheduled to be done on May 3rd between the time frame
(0000-0500) if we determine there is no problem to do so.

Once the power is stabilized back to UPS-A1&2, further investigation into the root
cause of this outage can be completed and documented for notice and any additional
appropriate remedial action needed. TELEHOUSE remains committed at all times to
providing you with up to date status on current conditions.

Sincerely yours,