Are power generators really up the task?
Recently Rackspace experienced an outage caused by their generators. I would not open this if its wasnt because in the last 10 years everytime there is a power outage on a DC the generators failed. (this is not a rackspace post, i just took 1 of 100 examples)
I dont want to mention names but I have seen plenty of DCs on which generators fail when the task is loaded to them. Some people said its because the maintaince is expensive and they dont do it on time.
Thats not always the case as I have seen DCs which made their maintainces on shape and time and they also had problems.
Now my question is if putting up generators costs so much money and we see them fail over and over again, are they up the task and worth the investment?
We know network only needs a short time without power and all is lost. This topic is ratter an approach on how to best handle power loads. Im sure there must be some way to actually be sure that when power FAILS, generators will keep up the task. The most sure testing would be to actually shift load to the generators every some months to test them but that would be plain Crazy on a production datacenter. I see that most outages have always the same guilty gear, generators...
Lets take another example, Gmail recently failed big time, because some Google DCs dont have chillers to keep the DC from getting hot (New Europe DC) for example. On hot days they shift the load to other DCs and turn the DC off. That of course failed too on a real scenario. Now Google is not a standard DC, but if they keep such a risky scenario in place they must have a sure way that power will keep up when the load is shifted and there is a big peak.
Will there be a better solution in the next 10 years for power or are we going to keep hoping generators dont fail exactly when we need them the most. Maybe we all should have UPS (plenty)on a rack to keep the power until the DC brings up generators, but thats useless if the network is affected as well and it always is.