Web Hosting Talk : Web Hosting Main Forums : Colocation and Data Centers : Peer 1 Seattle (Westin) Planned power outage. What?

[funkee]

So I got an email from peer 1 detailing a 30 minute planned power outage on Mar 16th " to replace the transformer and bypass panel feeding our Seattle 7th Floor Data Center."

"There is also a potential for short periods of power loss during a 90 minute testing phase that follows the 30 minute shut down."

I have never heard of intentionally powering down a data centre. Is this common? I would think that in a properly designed data centre one would have multiple power feeds and have the ability to work on them separately without a service interruption.

They go on to say "Due to the impact this will cause we are offering you the option to have a Data Center Technician power off their equipment prior to the 30 minute shut down window. You can then choose to have your equipment powered on after the 30 minute shut down or wait until the 90 minute testing is complete."

[andrewinet]

Is it common, sure. It's common to power everything down on one feed, once every 3 to 5 years, for inspection and cleaning purposes at the very minimum. That's why some facilities operate dual independent A/B feeds with shared-nothing architecture - if you need to take one down, the other one remains perfectly viable.

[Dougy]

Originally Posted by funkee

So I got an email from peer 1 detailing a 30 minute planned power outage on Mar 16th " to replace the transformer and bypass panel feeding our Seattle 7th Floor Data Center."

"There is also a potential for short periods of power loss during a 90 minute testing phase that follows the 30 minute shut down."

I have never heard of intentionally powering down a data centre. Is this common? I would think that in a properly designed data centre one would have multiple power feeds and have the ability to work on them separately without a service interruption.

They go on to say "Due to the impact this will cause we are offering you the option to have a Data Center Technician power off their equipment prior to the 30 minute shut down window. You can then choose to have your equipment powered on after the 30 minute shut down or wait until the 90 minute testing is complete."

I usually see a full shutdown with financial data centers since they are dead to the world on weekends.. but not normal datacenters..

[pubcrawler]

Sounds like 2011 is going to be the year of electric issues in data centers. Both intentional and unintentional.

[CGotzmann]

Originally Posted by funkee

So I got an email from peer 1 detailing a 30 minute planned power outage on Mar 16th " to replace the transformer and bypass panel feeding our Seattle 7th Floor Data Center."

"There is also a potential for short periods of power loss during a 90 minute testing phase that follows the 30 minute shut down."

I have never heard of intentionally powering down a data centre. Is this common? I would think that in a properly designed data centre one would have multiple power feeds and have the ability to work on them separately without a service interruption.

They go on to say "Due to the impact this will cause we are offering you the option to have a Data Center Technician power off their equipment prior to the 30 minute shut down window. You can then choose to have your equipment powered on after the 30 minute shut down or wait until the 90 minute testing is complete."

Uh, unless you yourself are also running A+B circuits with redundant power supplies in your servers, you are more than likely plugged into ONE UPS, ONE ATS, ONE BREAKER PANEL, etc.
If any of those need maintenance or experience an outage, there is absolutely nothing you can do to avoid downtime. Unless your datacenter specifically sold you some other type of agreement, then thats how it most likely is.

[funkee]

Originally Posted by CGotzmann

Uh, unless you yourself are also running A+B circuits with redundant power supplies in your servers, you are more than likely plugged into ONE UPS, ONE ATS, ONE BREAKER PANEL, etc.
If any of those need maintenance or experience an outage, there is absolutely nothing you can do to avoid downtime. Unless your datacenter specifically sold you some other type of agreement, then thats how it most likely is.

Shouldn't the UPS sit between all that stuff and your servers? If equipment upstream from the ups needs servicing then that shouldn't be a problem and if the UPS needs servicing it can be bypassed temporarily.

I've had space in three data centres for almost 10 years and have had notifications of all kinds of electrical work but it has always been not service impacting.

FYI Luckily I don't actually have any equipment @ peer1 so this is just a philosophical discussion.

[pubcrawler]

@funkee

I agree with your assessment and share your opinion.

I find it shocking how non-redundant an ever growing number of data centers are in reality (not in their self promotion marketing materials).

When people come down on cheap facilities and DIY non traditional data centers it's for things like described - single homed non redundant power.

30 minute shutdown + 90 minutes of testing = 2 HOURS of downtime.

That would take that facilities uptime for the entire year (absent any other outages) down to 99.97% roughly.

Peer1 boasts a 100% uptime SLA seemingly standard. Obviously, this work window is some sort of exemption in the legalese I suspect.

There are going to be a number of folks who never got the email about this work. A number of servers that don't restart right or which services don't start and config issues. There always are

FYI: I don't have gear with Peer1 and won't any time in the future.

[CGotzmann]

Originally Posted by funkee

Shouldn't the UPS sit between all that stuff and your servers? If equipment upstream from the ups needs servicing then that shouldn't be a problem and if the UPS needs servicing it can be bypassed temporarily.

I've had space in three data centres for almost 10 years and have had notifications of all kinds of electrical work but it has always been not service impacting.

FYI Luckily I don't actually have any equipment @ peer1 so this is just a philosophical discussion.

The UPS can power equipment for at maximum between 5-10 minutes depending on load levels. UPS's are just used for the temporary interruption of power from utility feed drop to generator spinup/power up.
Though they said transformer so they could possibly bypass over to generator power while they perform the maintenance on the transformer... but then they also mentioned the bypass panel, which could mean the ATS bypass panel, thus rendering the ability to actually transfer between utility and generator power, null.

It happens... sometimes things just have to go down or break on their own, no matter how redundant

[andrewinet]

Originally Posted by funkee

Shouldn't the UPS sit between all that stuff and your servers?

Not necessarily. There are stepdown transformers ahead of the UPS at some point, but the UPS output power usually runs through another transformer, because it's usually 3ph 480v and needs to step down to something usable by customers, like 3ph 208v. As they mentioned the bypass panel (presumably the UPS full-wrap bypass), I'm guessing the transformer they're referring to is the output transformer.

Originally Posted by pubcrawler

When people come down on cheap facilities and DIY non traditional data centers it's for things like described - single homed non redundant power.

Ok but there's nothing wrong with a facility owner choosing to operate this way, and there's nothing wrong with a customer buying this service as-is, so long as the facility is honest and forthright about their capabilities. Some people are ok paying double for A and B feeds, usually totaling some $750/mo for dual 110v 20a circuits. Others want to pay what this actually costs the facility, typically a bit under $200/mo for a single feed, and are willing to forego the increased reliability along with the price. Multiply that over 10 or 20 cabinets.. the difference is a Lexus and a big house.

[IRCCo Jeff]

We were down for about 45 minutes in 2009 when Peer1 lost an entire row of cabinets in Los Angeles. They didn't know about it until we opened a ticket and someone went to look at it.

At least this time you have warning

[appliedops]

Originally Posted by funkee

I have never heard of intentionally powering down a data centre. Is this common? I would think that in a properly designed data centre one would have multiple power feeds and have the ability to work on them separately without a service interruption.

It happens quite a bit, its called "maintenance"
Seriously though, everything has to turn off sometimes, one way or another.

[Rob T]

There are certainly issues where power has to be disabled in order to perform upgrades, maintenance, or to make other changes. Based on the info provided by Peer1 in the email you shared, I'd guess that they may be doing maintenance on a low-voltage distribution transformer that feeds a distribution panel and it's related bypass - because those are downstream of any UPS gear in most cases, it's not typically possible to perform maintenance on that type of gear without taking the power offline. That's just a guess, but it would make sense as to why they can't maintain power during the maintenance.

Even in facilities that offer a true A/B power infrastructure, in my experience many of the customers in the facility either won't elect to pay for A/B power, and those that do in many cases make implementation errors that result in a partial outage if one of their power feeds does go down. We typically recommend that our customers utilize a rack PDU that has a built-in ATS to make sure that they are truly A/B redundant, that way they don't have to use servers and other equipment with dual PSU's in order to maintain A/B power redundancy at the equipment level.

[andrewinet]

To the original poster, I just noticed the "bad design" tag that you placed on this thread. You did know you were buying a single feed, and that invariably you'd have no redundancy *when*, not *if* they needed to take the power down for maintenance, right?

[leeware]

I was going to reply to this after he first posted but it is another example of people (a) not understanding what they are buying and (b) having completely unrealistic expectations of services when it comes to the prices they are paying.

As others have alluded to here, in a shared nothing environment where the cost would make most users lurking on WHT puke these kinds of issues are less common. However, for just about any service this is affordable to people buying services offered here, there is a good chance these things will happen from time-to-time. This doesn't mean that a facility is poorly design. Providers can spend endless amounts of money trying to make sure everything is always available. However, the fundamental problem is twofold:

(1) Most customers are not going to want to pay the higher costs for such reliability ($$$-$,$$$/month).


(2) It seems rather pointless to complain about a providers lack of redundancy especially when customers demand services for pennies and don't invest their savings into building redundancy into their applications or services.


Therefore, the approach most end-users take is: Find least expensive provider that offers the highest levels of reliability and host my single (point-of-failure) application with said provider. When something goes wrong, point the finger at the provider.

It would be helpful if these users remembered the motto:

Two is one and one is none.

[RSanders]

Sure, 440V UPS to transformer to service panel. This sounds like a typical setup and the only way to service it past the UPS point is to shutdown.

This is why redundant power paths can be important. You should be able to order a second power circuit off of a independent UPS and transformer, at a premium price. This can not be built into a single circuit feed and is much more expensive to provide.