Results 1 to 29 of 29
  1. #1
    Join Date
    Jun 2005
    Posts
    3,448

    Are power generators really up the task?

    Recently Rackspace experienced an outage caused by their generators. I would not open this if its wasnt because in the last 10 years everytime there is a power outage on a DC the generators failed. (this is not a rackspace post, i just took 1 of 100 examples)

    I dont want to mention names but I have seen plenty of DCs on which generators fail when the task is loaded to them. Some people said its because the maintaince is expensive and they dont do it on time.

    Thats not always the case as I have seen DCs which made their maintainces on shape and time and they also had problems.

    Now my question is if putting up generators costs so much money and we see them fail over and over again, are they up the task and worth the investment?

    We know network only needs a short time without power and all is lost. This topic is ratter an approach on how to best handle power loads. Im sure there must be some way to actually be sure that when power FAILS, generators will keep up the task. The most sure testing would be to actually shift load to the generators every some months to test them but that would be plain Crazy on a production datacenter. I see that most outages have always the same guilty gear, generators...

    Lets take another example, Gmail recently failed big time, because some Google DCs dont have chillers to keep the DC from getting hot (New Europe DC) for example. On hot days they shift the load to other DCs and turn the DC off. That of course failed too on a real scenario. Now Google is not a standard DC, but if they keep such a risky scenario in place they must have a sure way that power will keep up when the load is shifted and there is a big peak.

    Will there be a better solution in the next 10 years for power or are we going to keep hoping generators dont fail exactly when we need them the most. Maybe we all should have UPS (plenty)on a rack to keep the power until the DC brings up generators, but thats useless if the network is affected as well and it always is.

  2. #2
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,889
    The issue is general not with the generators themselves, but with the transfer switches or control systems in charge of the generators, at least that was the issue at Equinix Chicago a couple years ago. The issue you'll run into is a single point of failure somewhere in the system, outside the generator, and those types of issues will still be there even with a power source other than the generator. You're blaming the generators for basically all power sub-system failures, so looking at it that way, of course the "generators" are going to be the most common point of failure.

    In addition, you don't have a good sample size. How many times are generators used that do their job? I'd say on average a facility will go to generator 1-2 times a year, yet you're not seeing every facility having a power failure 1-2 times a year. For the most part, when a facility goes to generator, you're not even going to notice.

    Finally, what should be done other than a generator? There is no other solution I can think of, other than simply having the battery power to potentially ride out a couple day outage, which would be obscenely expensive and a major maintenance hassle. Yes, a generator is expensive, but it should take you from ~99.9% to ~99.99% reliability. For most colocation facilities, that extra reliability is easily worth the cost, though for others it isn't, which is why facilities like FDC exist, etc.
    Karl Zimmerman - Steadfast: Managed Dedicated Servers and Premium Colocation
    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation
    Now Open in New Jersey! - Contact us for New Jersey colocation or dedicated servers

  3. #3
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Quote Originally Posted by KarlZimmer View Post
    The issue is general not with the generators themselves, but with the transfer switches or control systems in charge of the generators, at least that was the issue at Equinix Chicago a couple years ago. The issue you'll run into is a single point of failure somewhere in the system, outside the generator, and those types of issues will still be there even with a power source other than the generator. You're blaming the generators for basically all power sub-system failures, so looking at it that way, of course the "generators" are going to be the most common point of failure.

    In addition, you don't have a good sample size. How many times are generators used that do their job? I'd say on average a facility will go to generator 1-2 times a year, yet you're not seeing every facility having a power failure 1-2 times a year. For the most part, when a facility goes to generator, you're not even going to notice.

    Finally, what should be done other than a generator? There is no other solution I can think of, other than simply having the battery power to potentially ride out a couple day outage, which would be obscenely expensive and a major maintenance hassle. Yes, a generator is expensive, but it should take you from ~99.9% to ~99.99% reliability. For most colocation facilities, that extra reliability is easily worth the cost, though for others it isn't, which is why facilities like FDC exist, etc.
    I agree with you 100% Karl.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  4. #4
    Join Date
    Sep 2008
    Location
    Dallas, TX
    Posts
    4,552
    Quote Originally Posted by KarlZimmer View Post
    The issue is general not with the generators themselves, but with the transfer switches or control systems in charge of the generators, at least that was the issue at Equinix Chicago a couple years ago. The issue you'll run into is a single point of failure somewhere in the system, outside the generator, and those types of issues will still be there even with a power source other than the generator. You're blaming the generators for basically all power sub-system failures, so looking at it that way, of course the "generators" are going to be the most common point of failure.

    In addition, you don't have a good sample size. How many times are generators used that do their job? I'd say on average a facility will go to generator 1-2 times a year, yet you're not seeing every facility having a power failure 1-2 times a year. For the most part, when a facility goes to generator, you're not even going to notice.

    Finally, what should be done other than a generator? There is no other solution I can think of, other than simply having the battery power to potentially ride out a couple day outage, which would be obscenely expensive and a major maintenance hassle. Yes, a generator is expensive, but it should take you from ~99.9% to ~99.99% reliability. For most colocation facilities, that extra reliability is easily worth the cost, though for others it isn't, which is why facilities like FDC exist, etc.
    Does FDC not use generators?

  5. #5
    Join Date
    Nov 2003
    Location
    Lynnwood, WA
    Posts
    438
    I see data centers go on generator power literally every week or two due to spottiness of grid power; 100's of times over a few years, they do this, and you never hear about it, because it just works.

  6. #6
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,889
    Quote Originally Posted by Jacob Wall View Post
    Does FDC not use generators?
    In Chicago, correct.
    Karl Zimmerman - Steadfast: Managed Dedicated Servers and Premium Colocation
    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation
    Now Open in New Jersey! - Contact us for New Jersey colocation or dedicated servers

  7. #7
    Join Date
    Jun 2005
    Posts
    3,448
    Quote Originally Posted by KarlZimmer View Post
    In Chicago, correct.
    What do they use?

  8. #8
    Join Date
    Aug 2006
    Location
    London
    Posts
    549
    @nibb,

    As they're in the The Chicago Board of Trade (CBOT) building which is apparently the only building in Chicago to draw power from 6 different city substations, they say its already fully redundant.
    Ekin Ersoy
    Stagnom Servers | Europe Dedicated Servers with Redefined Support
    Follow us on Twitter: @StagnomUK | Like us on Facebook: facebook.com/Stagnom

  9. #9
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,301
    Quote Originally Posted by Ekin View Post
    @nibb,

    As they're in the The Chicago Board of Trade (CBOT) building which is apparently the only building in Chicago to draw power from 6 different city substations, they say its already fully redundant.
    Unless they need to shut down power to the whole building FWIW, I don't believe their Denver facility has any sort of UPS either.

    Beyond that, in regards to power, the only way to ensure you have reliable, redundant power is to feed your servers off 2 completely separate power systems (2N or 2N+1). Downtime (planned and unplanned) can and will happen within the confines of a single electrical system. Just because a single system has some redundancies built into it, does not mean that system is designed or capable of handling every kind of failure, or multiple failures of sub-systems within that occur at the same time. Beyond that, the more redundant a system becomes, the more complicated it becomes ... increasing the likelyhood of some sort of human error creating downtime because they do not understand how the system is designed to operate.

    And generators are not the only source of issues. UPSs are also, ironically, the source of a lot of downtime. Batteries venting, modules blowing up, faulty control systems deciding the system is in overload and dropping the load.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  10. #10
    FWIW the only alternative to a generator I can think of would be a fuel cell but that probably wouldn't be cost effective and it's unlikely you'll be able to get hydrogen refills easily.

    I don't know much about fuel cells but I believe they may have a few advantages over traditional diesel generators. I'm pretty sure they take up less space although tanks may have to be made larger and they have fewer moving parts to fail (none?). However most importantly it sounds cooler than the run of the mill generator system .

  11. #11
    Join Date
    Apr 2006
    Location
    Phoenix
    Posts
    808
    Quote Originally Posted by Jay Suds View Post
    Beyond that, the more redundant a system becomes, the more complicated it becomes ... increasing the likelyhood of some sort of human error creating downtime because they do not understand how the system is designed to operate.
    Jay,

    This is actually something I speak on quiet a bit, people often get confused between redundancy and reliability.
    Jordan Jacobs | VP, Products|SingleHop| JJ @SingleHop.com
    Managed Dedicated Servers | Bare-Metal Servers | Cloud Services

  12. #12
    Join Date
    Aug 2002
    Location
    Atlanta, GA
    Posts
    1,114
    Beyond that, in regards to power, the only way to ensure you have reliable, redundant power is to feed your servers off 2 completely separate power systems (2N or 2N+1).
    That alone is not enough. I know of one large data center here that claims all that stuff. But last time I checked they still had several single points of failure including it all still went to a single transformer before entering the building. (if they really even had multiple power feeds?)
    SiteSouth
    Atlanta, GA and Las Vegas, NV. Colocation

  13. #13
    Join Date
    Apr 2006
    Location
    Phoenix
    Posts
    808
    Michael,

    That is the reason I always suggest to demand a power map from your facility before signing a contract.
    Jordan Jacobs | VP, Products|SingleHop| JJ @SingleHop.com
    Managed Dedicated Servers | Bare-Metal Servers | Cloud Services

  14. #14
    Join Date
    Dec 2001
    Location
    Atlanta
    Posts
    4,419
    Other issues to look at is in a severe disaster who can get fuel and how much fuel does your dc have on site to weather the issue as well as what priority are they on the critical infrastructure restoral list. the higher they are the faster they can get restored ahead of others and can get fuel before others.

    I hear people talk about fuel delivery contracts but these are bogus in a real serious event since everyone will call for fuel at the same time and then it will be who is the critical need vs who has a contract (which I have never seen anyone be able to produce one - I think that these are mythical beasts talked about by sales teams).

    I can tell you though the power company does have a list of critical infrastructure requiring first priority on restoration.
    Dedicated Servers
    WWW.NETDEPOT.COM
    Since 2000

  15. #15
    Join Date
    Dec 2004
    Location
    Butler,TN
    Posts
    2,413
    Hi!
    Ah! interesting..I was going to start a thread on this. Thank you..you saved me some time.

    6 different substations indeed. I remember a NOC bragging about being fed by two different stations..guess what..it went down..for a good long time too. Blah. So much for those two feeds providing redundancy. It didn't happen.

    I think the problem is you have you have people designing these systems who have zilch expertise in said area. Period.

    And when (if) they test the generators..they do so with no load. I've seen it first hand. Yes..no load at all. Nice.

    Let's take a look at Rackspace's recent downtime..shall we?

    What Happened?

    "Each generator failed on a loss of excitation – an inability to maintain the magnetic field. But it was really the inability to get synchronized that created that fault.”

    Could it have been prevented?

    Yes.

    http://www.gedigitalenergy.com/multilin/catalog/g60.htm

    They cost a little more than $10K base.

    There you go. Oh..one of these babies can handle up to six generators.

    I am sorry to say it can't make a good cup of coffee, though.

    >>>Legal disclaimer: I am no engineer..I am just good at finding things that solve problems.

    Bryon
    Bryon L Harvey
    Soil Relocation Engineer

  16. #16
    Join Date
    Jun 2009
    Location
    California
    Posts
    509
    When I was doing commo in the Army that is exactly what we did, live tests of the battery/generator systems once a month. Shutting off commercial power, watching the batteries kick in, bringing generator power online.

    I don't know why you wouldn't want to do that, unless you don't have any confidence in your system.

  17. #17
    Join Date
    Dec 2001
    Location
    Atlanta
    Posts
    4,419
    Quote Originally Posted by Annuit Cúptis View Post
    When I was doing commo in the Army that is exactly what we did, live tests of the battery/generator systems once a month. Shutting off commercial power, watching the batteries kick in, bringing generator power online.

    I don't know why you wouldn't want to do that, unless you don't have any confidence in your system.

    I dont know why most dont. its what the manufacturers recommend.
    Dedicated Servers
    WWW.NETDEPOT.COM
    Since 2000

  18. #18
    Join Date
    Jul 2009
    Posts
    240
    back in my AT&T days ..our MSC generators were run every thursday ..power was switch from the grid ..to batteries ..to generator for 1 hour every week. Maybe its just economics for some DC operators - it cost a lot to maintain and operate a generator system. Last project there was spending over $20M creating a new MSC ..most of that money went to the cooling and generators/UPS system.

    Heck ..one of the biggest telco hotel in Los Angeles dont even run their generators monthly ..i think maybe 1 or 2 a year if that.

  19. #19
    Join Date
    Jun 2004
    Location
    Hyderabad India
    Posts
    1,103
    It is always better if there is A+B entirely independent power system with own switching gear etc.. + each equipment with RPS. even if one side has to fail entirely fire or transfer failure the other side should not be effected in any way.

    Also how about maintaining 10 small UPS systems 4 small generators to back up with, 2 transformers to feed then whole DC does not have to shutdown just because something failed may be part of it. Network gear can have additional independent UPS in rack as if that has to fail there is no point in keeping systems online

  20. #20
    Join Date
    Jun 2004
    Location
    Hyderabad India
    Posts
    1,103
    I remember one time due to chiller failure DC had to shut down systems to keep temperature in control, if there were multiple small cooling units this may not have happened, say 20% of units to fail temperature may have increased a bit but should not have effected entirely,

  21. #21
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Quote Originally Posted by DJMizt73 View Post
    back in my AT&T days ..our MSC generators were run every thursday ..power was switch from the grid ..to batteries ..to generator for 1 hour every week. Maybe its just economics for some DC operators - it cost a lot to maintain and operate a generator system. Last project there was spending over $20M creating a new MSC ..most of that money went to the cooling and generators/UPS system.

    Heck ..one of the biggest telco hotel in Los Angeles dont even run their generators monthly ..i think maybe 1 or 2 a year if that.
    That is truly sad and from what I remember in the years I had equipment in LA I really do believe it.

    Imagine how (un)reliable your car would be if you fired it up once or twice a year at most and just drove it around the block once? Especially after many years?
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  22. #22
    Join Date
    Sep 2004
    Location
    San Jose
    Posts
    66
    Quote Originally Posted by DJMizt73 View Post
    back in my AT&T days ..our MSC generators were run every thursday ..power was switch from the grid ..to batteries ..to generator for 1 hour every week. Maybe its just economics for some DC operators - it cost a lot to maintain and operate a generator system. Last project there was spending over $20M creating a new MSC ..most of that money went to the cooling and generators/UPS system.

    Heck ..one of the biggest telco hotel in Los Angeles dont even run their generators monthly ..i think maybe 1 or 2 a year if that.

    In a lot of places (California is especially bad), there are restrictions in place for how many hours a year a generator can be run for testing. 5-20 hours a year is fairly average in California, with stiff fines for going over that. This is for testing only. Real power outages they can be ran as long as needed.

  23. #23
    Join Date
    Jul 2009
    Posts
    240
    Quote Originally Posted by ST-3 View Post
    In a lot of places (California is especially bad), there are restrictions in place for how many hours a year a generator can be run for testing. 5-20 hours a year is fairly average in California, with stiff fines for going over that. This is for testing only. Real power outages they can be ran as long as needed.
    Gotta thank the CARB for that

    I still remember that one incident when they try to switch to the genset and knocked 1/2 of the building's power (the LA Telco hotel) - we were on batteries for I cant even remember how long, we squeezed every juice out of that enegizer bunny

  24. #24
    Join Date
    Dec 2000
    Location
    The Woodlands, Tx
    Posts
    5,962
    Quote Originally Posted by sailor View Post
    Other issues to look at is in a severe disaster who can get fuel and how much fuel does your dc have on site to weather the issue as well as what priority are they on the critical infrastructure restoral list. the higher they are the faster they can get restored ahead of others and can get fuel before others.
    Good point, and one I will jump on. When the hurricane came thru Houston and knocked out power for two weeks, we stayed up, thanks to our backup power.... which runs on propane, and so do the delivery trucks. Our tank contains enough fuel to last two weeks, we had a truck sitting in the yard within 24 hours ready to refill if/when needed.... which we didnt.

    Not to say we weren't completely fuc!ing worried.... generators really arent designed to worked that long. It's like putting a brick on your gas pedal in your car for two weeks.... in the end, a generator is still just an engine. Belts can go, pullies can go, oil can lose its viscosity and lock the motor up... but we faired well.

    Quote Originally Posted by Annuit Cœptis View Post
    When I was doing commo in the Army that is exactly what we did, live tests of the battery/generator systems once a month. Shutting off commercial power, watching the batteries kick in, bringing generator power online.
    It's what we do once a month. We pull that great big lever on the side of the building. We are fully on alternate power within 4 seconds. Some will find that hard to believe, so maybe I will video the next test

    A lot of places rely on the maintainence contractors to assure them everything will work in a failure. We made this mistake when we built the place. They did load tests, everything checked out fine. When we finally got gutsy enough to do a live test, we failed. Something about the field or frequency, or something. Batteries when crazy flashing, beeping, clunking.

    So now we manually live test once a month, not that our system isnt already well tested. Grid power sucks here, and we get tested by the electric company several times a month.. Who needs to "manually" test when you have ENTERGY as your electric company! lol. Needless to say that was a VERY scary test.

  25. #25
    Join Date
    Mar 2009
    Location
    $HOME
    Posts
    85
    Quote Originally Posted by Annuit Cœptis View Post
    When I was doing commo in the Army that is exactly what we did, live tests of the battery/generator systems once a month. Shutting off commercial power, watching the batteries kick in, bringing generator power online.

    I don't know why you wouldn't want to do that, unless you don't have any confidence in your system.
    Running all gear from battery for 10 seconds every month... do you need to de-rate the life expectancy of UPS batteries due to all the hits to battery?

    To be nice to the batteries, I use the ATSes' "exercise with load" feature, which starts generators first, before the ATSes phase-sync and transfer. The cut-over is fast enough that some diagnostic equipment that is on ATS but doesn't reboot.

    I open the mains yearly, which does hit the batteries for ten seconds.

  26. #26
    Join Date
    Jul 2002
    Location
    Atlanta, Georgia
    Posts
    37
    We have just completed commissioning a new data center and we load tested the whole center in different conditions off and on for two weeks. The manufacturer told us to plan to work the generators hard a few times per year to maximize their life (plus it serves as a nice sanity check for us, as well).

    Has anyone else been approached by their utility with offers of a fairly large rebate (six figures per year) in exchange for switching to the gen sets for a few hours per month in heavy load times (mid summer afternoons and such)?

    Just curious.


    --T

  27. #27
    Join Date
    Apr 2006
    Location
    Phoenix
    Posts
    808
    Quote Originally Posted by SweetT View Post
    We have just completed commissioning a new data center and we load tested the whole center in different conditions off and on for two weeks. The manufacturer told us to plan to work the generators hard a few times per year to maximize their life (plus it serves as a nice sanity check for us, as well).

    Has anyone else been approached by their utility with offers of a fairly large rebate (six figures per year) in exchange for switching to the gen sets for a few hours per month in heavy load times (mid summer afternoons and such)?

    Just curious.


    --T
    Tony,

    They contacted us and we are working through it now, they call it demand response, http://www.enernoc.com/. As long as you have Tier 2 generators its a pretty good system, the only drawback is if you have an open transition ATS - you would roll to batteries every time you switch from Utility to Gen and back again.

    Even with an open transition ATS however, its well worth it IMHO.

    Can't wait to fly out there and check out the new facility! Congrats!
    Jordan Jacobs | VP, Products|SingleHop| JJ @SingleHop.com
    Managed Dedicated Servers | Bare-Metal Servers | Cloud Services

  28. #28
    Can you say closed transition switchgear. With this type equipment, the load does not hit the batteries as the transition to and from power company in live load test is overlapping between the generators and the power feed. Also, with tier 2 generators, they are standby rated which means they are not rated for prime generation; only in standby emregency situations.

  29. #29
    Join Date
    Apr 2006
    Location
    Phoenix
    Posts
    808
    Harry,

    Your 100% correct, I was simply saying in an open transition deployment one of the downfalls.

    Teir 2 is simply an EPA guideline and has no impact on standby or prime generation. That being said, you would normally de-rate your gensets if you were planning on running in prime.
    Jordan Jacobs | VP, Products|SingleHop| JJ @SingleHop.com
    Managed Dedicated Servers | Bare-Metal Servers | Cloud Services

Similar Threads

  1. URL / Name Generators
    By wbengal in forum Domain Names
    Replies: 8
    Last Post: 10-24-2005, 06:26 PM
  2. Form Generators
    By Russ Foster in forum Web Design and Content
    Replies: 1
    Last Post: 04-10-2005, 06:06 PM
  3. any generators?
    By Torith in forum Web Hosting Lounge
    Replies: 4
    Last Post: 04-28-2004, 09:37 PM
  4. Generators
    By RossH in forum Dedicated Server
    Replies: 17
    Last Post: 09-21-2003, 06:05 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •