Web Hosting Talk : Web Hosting Main Forums : Colocation and Data Centers : Power issues at data centers

[zweb]

Is the type of outage at MySpace.com this weekend (due to a generator failure) a
common occurance? It seems like this happeded about the same time last year.

Or is it just the datacenter in LA where MySpace has it's servers?

Is it odd that a site the size of Myspace is located all in one data center or is
that common?

[dkitchen]

Though power outages at datacentres should be a rare occurrence, they are becoming more and more common, and it seems as though more and people people are just "accepting this".

It is strange that MySpace is in a single datacentre and not globally load balanced with the likes of Google, Yahoo, etc, etc. Perhaps they will rethink after this outage, it will have cost them a small fortune.

Dan

[snickn]

My guess is this would be due to the fact the site was never built to be scalable. I'm guessing they are slowly redoing the whole site behind the scenes.

[HostTitan]

My understanding was that they were in multiple datacenters (for backup/reliablity) but that there are single points-of-failure in the way its been designed. Its tough to replicate the database and data in real-time such that an entire datacenter outage wouldn't be noticed.

[spaethco]

I think the problem with data center power is that there are just so many components to fail. From what I have seen generator failure is actually pretty rare, but transfer switch failure shows up as an issue more frequently than you'd think. (To use the generator you need to disconnect from the grid and feed in from the generator) Of course, sometimes you just get slammed with bad luck. We had a DC lose power in one of the server rooms because the UPS took itself offline for a scheduled 45-second self test ... and during that self-test commercial power failed. According to the UPS manufacturer we are the first company to ever have this happen.

The other issue is that even with really good UPS back-up systems, if there is any delay in getting the generator started you could still lose the site. Ironically UPS runtime generally isn't the issue, but rather the cooling systems for the rooms usually require site power to function. We had an issue this year at one of our smaller DCs where we lost 1 of 2 commercial power feeds, the transfer switch didn't transfer the load to the secondary feed but it did detect that the secondary feed was energized so it also didn't kick on the generator. All our systems stayed up on UPS power, but once room temperatures made it up to 125F we had to do an emergency power-off to the rooms anyway.

Overall the issue is that backup systems just don't get tested enough. At the last place I worked they went as far as having all servers with dual power supplies and dual PDU feeds, and every time you installed hardware they would shut off a PDU feed to the entire rack on each side to make sure the load could be sustained. Seems drastic, but we always did installs off-hours so the impact would be minimized if a breaker would trip. Where I work now, for our flagship datacenters we have agreements with the local power companies to go "off-grid" during peak energy times, so during the summer we run off generator 2-3 days per week. We still carry out regular testing through the winter and make sure to do a generator run at least every 2 weeks. As our company continues to grow through acquisitions, it becomes very clear which companies installed backup systems but never/rarely tested them... they tend to be the places we have the most issues.

Eric

[DomainWorks]

I don'' think we know enough about the particualrs of the MySpace incident to conclude much. But generators often fail because they haven't been regularly tested or maintained. In long-term outages fuel can be a problem, although it doesn't seem like the grid outages would have brought that into play.

[lockbull]

Just out of curiosity, was this a private data center? If not, anyone know which facility it was?

[amps]

Myspace is in 1200 W. 7th street, the Garland building, based on the power issues they experienced last year.

[mdrussell]

I was in One Wilshire on Friday and everything looked fine. I don't think others in One Wilshire were affected, sounds like an excuse to me.

[jwr]

Wasn't there a fire in one wilshire just a few days ago?

[AlieN]

It IS becoming more and more common.

When people learn that you can't take an office building and "convert" it into a datacenter, these power issues may be mitigated.

Until then you will find datacenters on single power grids, insufficient cooling and with insufficient maintenance schedules. We got burnt many many years ago with something like this, and moved shortly afterwards.

At a colo you should be paying for "2" primary things, Space and Power. When one of these 2 things fail..it's time to jump ship.

The problem is that when it "gets too hot" or the power goes out...there is no money on earth that can/will save face with your customers...and the colo, 9/10 doesn't care..you will be paying the bill the next month.

My 2c.

Anton
Webhosting.net, Inc.

[Dennis Nugent]

Everyone wants cheap power... until there is a power outage.
As discussed numerous times over the last two years, data center "A" might charge $200 for 20 amps, and data center "B" might charge $100 for 20 amps.
A good chunk of that $200 probably goes for generator and UPS amoritization and maintenance and TRAINING, whereas the other data center prays that they dont have an outage.

There was a power outage at 200 Paul 2+/- years ago, on a weekend. Onsite staff walks out, yep, there is a power outage, generators are on, and records it in the log books like he has been trained to do. One of the generators canabilizes itself, and power goes out on a couple of floors. Onsite guy is still sitting there recording stuff in the log book; apparently never notified management.

You walk into some data centers and the amount generator/UPS redunancy, training, polices and procedures implentation, etc are apparent. Those are the data centers that you want to be in.

</rant>

[music]

One should always look at how many power grids feed the building/facility and building management/ownership. But like everything else it is give-and-take.

The public is use to having interruption on the internet so unless you are a super-mission-critical status (Most of WHT is probably not) than a brief interruption will not put you out of business. It comes back to the facility/building management having the tools and resources to fix problems quickly and a proactive maintenance policy.

[RossH]

Every data center will have an issue at one time or another no matter how well they plan. What you need to look at it how they react during crisis time and frequency of the problems

[cabalstudios]

Talking about outages - http://www.theregister.co.uk/2006/07...unday_failure/