Results 1 to 8 of 8

Thread: Load Testing

  1. #1

    Load Testing

    We have been doing some research on some colocation facilities lately and a question came up about load testing. I'm wondering what the standard is in regards to load testing a colocation facility.

    Are they full load tests? How often are they done? How long are those tests done for?
    FXH, Inc :: Web Hosting, Web Design, rsync and R1Soft Backup Solutions, and more!

  2. #2
    I assume that you're talking about generator load-tests. Different techs run their generator tests differently, but I think that a full load test should be performed at least yearly as part of generator preventative maintenance. If the genset frequently runs at light loads it might be necessary to perform more often.

    Diesels will build up carbon in the engine when run at light load. The load bank test (which is essentially a big hair-dryer) will fully load up the engine and allow it to burn off the excess carbon deposits.

    Ed

  3. #3
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,302
    We run a load bank test on our genset weekly for 30 minutes. We do a full load transfer test quarterly and we simulate utility power failure annually. Those tests are also 30 minutes in duration. We also have our genset vendor do PMs monthly.

    Genset, cooling and UPS maintenance is actually one of those things that's going to vary quite a bit from data center to data center. If a data center wants to be cheap, they will try to get away with doing these things not as often as they should be, and the end result will be an unexpected outage of a critical infrastructure system. As a paying colocation customer, you are more than justified asking any potential colocation provider how often they do maintenance on their systems and to requested copies of associated documentation.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  4. #4
    Thanks for the detailed response Jay. I didn't get as much as a response as I expected, but the information helps.

    From what I have been seeing, simulated utility power outages is going to be an annual thing. As well as some tests done about once per quarter. I assume that tests like these prove to be enough?

    Most of the datacenters that I have received information from haven't had an issue presenting the information. I never thought of asking for the documentation to back it.

    Thanks,
    Ryan
    FXH, Inc :: Web Hosting, Web Design, rsync and R1Soft Backup Solutions, and more!

  5. #5
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,302
    FWIW, if we didn't have the load bank, we would be doing monthly load test transfers. The weekly load bank tests really keep the unit in ready to go condition. Running the generator is really the key piece. Obviously, it's important for your ATS to be in good working condition, but there's so much that can go wrong with the generator, and by not running and servicing it regularly you run the risk of developing a lot of problems and being totally unaware of them.

    Interestingly enough, the space we took over was unoccupied for 3 years. The generator didn't have a PM during this time, but the load bank test was running weekly. When we had it serviced the first time, it was in nearly perfect working condition and nothing non-routine had to be replaced or adjusted. Had the unit sat for 3 years, it probably would have been a very different story.

    As for simulated utility outages, our critical power vendor recommend that those only be done yearly. Based upon his recommendation, my assumption is that that's a pretty standard thing, and acceptable in many different critical fields. In particular, he noted his hospital clients did year simulated utility outages.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  6. #6
    Join Date
    Nov 2004
    Location
    Atlanta, GA
    Posts
    464
    Quote Originally Posted by Jay Suds View Post
    Interestingly enough, the space we took over was unoccupied for 3 years. The generator didn't have a PM during this time, but the load bank test was running weekly. When we had it serviced the first time, it was in nearly perfect working condition and nothing non-routine had to be replaced or adjusted. Had the unit sat for 3 years, it probably would have been a very different story.

    As for simulated utility outages, our critical power vendor recommend that those only be done yearly. Based upon his recommendation, my assumption is that that's a pretty standard thing, and acceptable in many different critical fields. In particular, he noted his hospital clients did year simulated utility outages.
    The sitting also depends greatly on the amount of oil & fuel in the tanks and the external environment. I was talking to someone that had a cat sit for 18 months in a very harsh environment and it fired right up.

    Our generator runs every Monday morning at 8am for a cool to full temp to cool down cycle. We do a load test monthly and drop the breaker quarterly. When I was working with some larger telecoms we had one facility that pulled the breaker every Friday at noon.

    I think that with the recent failures we have seen the poof is that people not doing simulated failures often enough is the primary reason for the failure.

    Jay, I must applaud you on the full simulation. There are TOO many datacenters that won't even transfer the load unless they have to. We recently had to deal with a failure mode that would only rear its head when under the facilities load and only happened when a certain pair of AC units compressors kicked in at almost the same time or a bit staggered.

    A funny that i recently heard from a generator tech that does an oversight PM for us said that one of the more "famous" facilities he works on will not even allow him to have the breaker to the generator on when he starts it to bring it up to temp!
    Linn Boyd

  7. #7
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,302
    We do full simulation because I need to sleep at night We are very luck to have great, stable power in downtown Denver, but eventually the utility power will inevitably fail. It doesn't make sense to have a generator, have an ATS, have a UPS, pay to maintain them, but then never actually simulate a full on utility failure. All of that stuff costs big money, and I want/need to know everything is working properly. And if something isn't working properly, I want to find out when we do the test with our critical power vendors on site, so they can respond and react to the situation instantly.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  8. #8
    Join Date
    Nov 2004
    Location
    Atlanta, GA
    Posts
    464
    Jay, tell me about it, it just amazes me that people are so quiet about these. We have just won some VERY large business because we would 1, give a demo run of the genset's and 2, allowed technicians onsite for the load test! :-D I do envy you though, I wish that we could practically have a full load bank, as i like to watch them scream!

    You know, it is nice when you have a 10 hour power outage facing you and you really don't care, with the exception of having to call the fuel trucks in the morning. I was at the facility for startup and then left to have a couple of glasses of wine, there were 0 issues on either side of the power for the facility.

    One other thing that is critical, ask the facility what their fuel storage, testing and fill specifications are. If the facility allows a fuel tank to go below about 95% full unless under an outage condition they are just asking for problems. The official word is 80% on smaller tanks, and 90% on larger ones, but on ones our size if we burn 5% we call for fuel.
    Linn Boyd

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •