Page 1 of 2 12 LastLast
Results 1 to 25 of 33

Thread: 7kW GPU farm

  1. #1

    7kW GPU farm

    I own a GPU Farm currently consuming up to 5 kW. It is made of 3U nodes consuming 1100W each (9.2A @ 120V) at full load and continuously. I need to expand to at least 7 kW soon and am looking for colo hosts in the Los Angeles area. I would appreciate some expert advice...

    I have done some research, obtained some price estimates. First problem is that power in LA is expensive. The best offer I have is $800/mo for one cabinet with 2x20A.

    Now get this: I can't even put 2 nodes on one circuit because that would be 18.4A, and continuous loads should not exceed 80% of the circuit rating, therefore should not exceed 16A on a 20A circuit. Which means the above offer only allows me to put 2 nodes per cab, or 1 per 20A circuit.

    Because this is a ridiculous waste of power and space (only 18.4A used out of 40A, only 6U used out of 42U), I plan to downclock the GPUs to drop the power consumption of each node so I can put 4 per cab, or 2 per circuit. In fact this is what I am currently doing with some nodes.

    But still I don't like wasting my hardware resources by downclocking. The best would be 208V/20A circuits: 3 nodes per circuit amounts to 15.9A... perfect. However I found no colo willing to sell 208V at my scale (5-7 kW).

    Any suggestion?
    Who sells 208V?
    Another option would be a single 120V/50A circuit (4 nodes on it = 76%) but who sells 50A on a single circuit?
    Also do you guys think I can find a better price than $800/mo for 2x20A in LA?

    Thanks in advance.

  2. #2
    Join Date
    Mar 2003
    Location
    chicago
    Posts
    1,781
    i dont think you will do better then $800 for 2 20a

  3. #3
    Contact Coresite - you'll save a boatload of money on power because they use utility billing, i.e. per kWh. Also they'll give you the power in whatever increment that they have breakers for. I don't think there's any such thing as a 50a breaker for a single phase, but surely 30a 208v to a single cabinet will fly. You might also be charged an extra fee of some kind for the "additional space" they'll allocate for your order. By that, I mean that you're using more than the typical 150 watts per sqft. density, so they're sure to charge you for "more space". Coresite has two LA facilities.

  4. #4
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,710
    Due to paying for the extra space for the density, the cost will not be any different. Either way you're paying for the density. What matters is the total final cost, and all the power requires a lot of cooling.
    EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
    Bandwidth Billing | Inventory & Asset Management | Server Control
    Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management

  5. #5
    What kind of facility do you need to be in? Unfortunately, for $800 a month in a city like LA, you won't get much in the way of redundancy...if you even need that. Especially at the power consumption you plan on having. Have you considered moving the colo out of LA?
    GenesisAdaptive.com · IP Transit and Connectivity, Colocation, & Enterprise IT Services
    █ Available in 9 Metros, 3 Continents, 1/10/40/100G Ports.
    Follow us on Twitter @GenesisAdaptive!
    █ Phone: (309) 807-5261 · Email: leec @genesisadaptive.com

  6. #6
    Join Date
    Jun 2004
    Location
    Hyderabad India
    Posts
    1,103
    That's a lot of processing, look for any colo who say high density who provide more power per cabinet who will surely have 208v, if it's not bandwidth intensive, you can use UK/Asia providers where you can get 220V supply , some custom plans
    Last edited by Krazy; 01-31-2011 at 02:23 AM. Reason: extra info

  7. #7
    I have considered moving out of LA, especially because my workloads are not bandwidth intensive at all. I have looked at Portland, Oregon where power is inexpensive. But even there I have not been able to find 1 cab with 40A at less than $800. I would love to be proved wrong.

    CGotzmann: I have no problem with paying for the extra space if this helps my host meet his power density target... you are right all that matters is the final cost.

    ServerGurus: quite frankly I don't need much redundancy or anything... just power.

    Thanks for the info guys (andrewinet, etc). I'll keep looking but so far this offer in LA looks like the best.

  8. #8
    Quote Originally Posted by xmrb View Post
    I own a GPU Farm currently consuming up to 5 kW. It is made of 3U nodes consuming 1100W each (9.2A @ 120V) at full load and continuously. I need to expand to at least 7 kW soon and am looking for colo hosts in the Los Angeles area. I would appreciate some expert advice...

    I have done some research, obtained some price estimates. First problem is that power in LA is expensive. The best offer I have is $800/mo for one cabinet with 2x20A.

    Now get this: I can't even put 2 nodes on one circuit because that would be 18.4A, and continuous loads should not exceed 80% of the circuit rating, therefore should not exceed 16A on a 20A circuit. Which means the above offer only allows me to put 2 nodes per cab, or 1 per 20A circuit.

    Because this is a ridiculous waste of power and space (only 18.4A used out of 40A, only 6U used out of 42U), I plan to downclock the GPUs to drop the power consumption of each node so I can put 4 per cab, or 2 per circuit. In fact this is what I am currently doing with some nodes.

    But still I don't like wasting my hardware resources by downclocking. The best would be 208V/20A circuits: 3 nodes per circuit amounts to 15.9A... perfect. However I found no colo willing to sell 208V at my scale (5-7 kW).

    Any suggestion?
    Who sells 208V?
    Another option would be a single 120V/50A circuit (4 nodes on it = 76%) but who sells 50A on a single circuit?
    Also do you guys think I can find a better price than $800/mo for 2x20A in LA?

    Thanks in advance.
    I would find it hard to believe a single server pulls down 1100w unless it's got multiple graphics cards in it. You might try removing one card from a machine rather than downclocking them. Providers can also probably do 30a circuits for you for 50% more than the cost of a 20a circuit. You should be able to get two nodes onto a 30a circuit, which would be less wasteful than 1 per 20a.

    Power in LA definitely is expensive. Utah has some of the lowest power costs in the country if that's at all an option for you, you might look into salt lake city. Phoenix also has pretty low power costs and is a solid Tier-2 city as far as the internet backbone goes.

    As far as it goes, I think you'll have a hard time beating $800 for 2x20a in LA.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  9. #9
    Quote Originally Posted by Krazy View Post
    That's a lot of processing, look for any colo who say high density who provide more power per cabinet who will surely have 208v, if it's not bandwidth intensive, you can use UK/Asia providers where you can get 220V supply , some custom plans
    UK is really a terrible place to host something like that, sure you get 230v, but the power over there is obscenely expensive, easily 50-100% more than LA even, on a KWH basis.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  10. #10
    Of course my nodes have multiple GPUs. The 1100W have been measured at the wall outlet. In fact, some nodes have less efficient PSUs and even hit 1250W.

    Removing 1 or 2 GPUs from each node is an alternative to downclocking, which I am considering too...
    Last edited by xmrb; 01-31-2011 at 05:23 AM.

  11. #11
    Quote Originally Posted by xmrb View Post
    Of course my nodes have multiple GPUs. The 1100W have been measured at the wall outlet. In fact, some nodes have less efficient PSUs and even hit 1250W.

    Removing 1 or 2 GPUs from each node is an alternative to downclocking, which I am considering too...
    One thing to definitely consider if you've not done so already is getting an 80+ gold rated power supply. When using that much power, it's pretty much mandatory.

    Here's the newegg list of 1100+ watt power supplies that are 80+ gold (they're all under $300, and even though they're desktop supplies, I would expect that they should fit into a typical 3u rackmount case):

    http://www.newegg.com/Product/Produc...1W%20-%201500W

    Mind you, PSU's watt ratings are typically from delivered DC power, not wall power, so even if you're pulling 1100 watts from the outlet, a 1200 watt PSU should still have some room as the system is probably "only" drawing 1000 watts. On the other hand, PSU's are most efficient when you're using close to 50% of their rated capacity, so running bigger or redundant power supplies might be a good option here.

    What are the specs on the actual server itself? Does the CPU run close to capacity during calculations, or just the GPU? These days a single socket system can easily draw 100 watts or less, so the power you waste using 3 servers with 2 cards each (for example), instead of 2 servers with 3 cards each, could be made up by the fact that the PSU in each of those servers will be operating close to it's optimal range, making either option likely to have the same power efficiency.

    Are these the desktop series, ~$500 apiece kind of cards, or the higher end Tesla kind of models? If it's the higher end Tesla, most of the cost of the system should be tied up in the cards and not the servers, so again, going with more servers with fewer cards each shouldn't increase total costs that much, and will give you more flexibility to run closer to the power limits of the circuit.

    Now that I think of it, a lot of rackmount cases these days are perfectly capable of dual redundant or even triple redundant power supplies. If you're pulling 1100 wall watts, you're probably delivering 900-1000 watts to your equipment. Running dual redundant 1k watt PSUs would allow each PSU to deliver 500 watts to your system, putting each psu at the magic 50% utilization level, where efficiency is highest, reducing your power use potentially by around 5%. The other advantage here, is that you can use one outlet on one PDU / Circuit, and the other outlet on a different PDU / Circuit. This way, you could put the equivalent load of 1.5 servers on one circuit, by connecting both plugs on one server to a circuit, both plugs on a different server to a different circuit, and 1 plug each on a third server to each of the two circuits. Getting 3 servers onto $800 / mo sounds a lot better than fitting 2 servers onto $800 / mo.

    The obvious downside is, if one of the circuits trips or loses power, the system that's "dual redundant" on both circuits will put it's full load onto the other circuit, causing it to trip the breaker. So it's not the best solution in terms of redundancy, but if you're just looking to get done with X amount of work for the lowest cost, it's the best way to go in your power profile. At 9.2a for one server, 1.5 servers would be like 13.8a, well within the 16a acceptable continuous usage profile, with a little left over for switches or whatever.

    Of course, if power really is the biggest concern, then Utah, where industrial power rates are less than half of those in Los Angeles, is a good option. Or somewhere where you can use free cooling instead of air conditioning most of the year.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  12. #12
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,131
    Quote Originally Posted by cyberhouse View Post
    i dont think you will do better then $800 for 2 20a
    you are not going to find a 40A cab for 800/month more like 1050-1200/month.
    Yellow Fiber Networks
    http://www.yellowfiber.net : Managed Solutions - Colocation - Network Services IPv4/IPv6
    Ashburn/Denver/NYC/Dallas/Chicago Markets Served zak@yellowfiber.net

  13. #13
    Join Date
    Jan 2011
    Location
    UK
    Posts
    776
    Quote Originally Posted by funkywizard View Post
    UK is really a terrible place to host something like that, sure you get 230v, but the power over there is obscenely expensive, easily 50-100% more than LA even, on a KWH basis.
    UK power is roughly £0.12/kWh, that's around US$.019c. You'll also have power density issues in most DC's in the UK.

  14. #14
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,957
    You're having trouble finding 208v, where are you looking? 208v is pretty standard from everything I've seen. Most of our new cabinet installs have been 208v, though we aren't in LA. If your facility can't/won't offer 208v options they must have NO control over their infrastructure or don't know how to build out a facility, so look elsewhere. There MUST be someplace in the LA area that can do 208v, and as you said, 20A 208v seems to make the most sense for you.

    I would first make absolute certain though that you're using the most efficient power supplies possible, as you've said yourself, that can make a major difference, and now ~90% efficient power supplies aren't that expensive either.
    Karl Zimmerman - Founder & CEO of Steadfast
    VMware Virtual Data Center Platform

    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation

  15. #15
    Join Date
    Sep 2010
    Location
    Dallas, TX
    Posts
    234
    I am surprised that 208v is that hard to source in LA as well. Either your sales rep isn't asking the right questions to operations or you are looking at some very old data centers that lack the density to support it. At the very least, they should offer you the power in a cage environment (to help spread the power/sqft).

    If you do go to 208v, make sure your auxiliary devices (if you have any) are capable of 208v as well. That's often overlooked until it's too late.

  16. #16
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,302
    Quote Originally Posted by ColoUnlimited View Post
    If you do go to 208v, make sure your auxiliary devices (if you have any) are capable of 208v as well. That's often overlooked until it's too late.
    Well this is wise advice, it is typically much ado over nothing as the large majority of all computing devices made in the last decade have auto sensing power supplies.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  17. #17
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,710
    He said that he can't find a colo that will do 208v at his density 5kw-7kw, not that 208v was not available.
    EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
    Bandwidth Billing | Inventory & Asset Management | Server Control
    Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management

  18. #18
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,957
    Quote Originally Posted by CGotzmann View Post
    He said that he can't find a colo that will do 208v at his density 5kw-7kw, not that 208v was not available.
    Interesting, pretty much the smallest 208v install we do is 30A 208v, which is 5kw, and the 2nd most popular is 2x 20A 208v, which is ~6.7kw. We can do that out of our oldest cage at 350 E Cermak which is ~10 years old at this point, so it sin't anything that new or interesting. Doing 5kw in a single cabinet should be EASY for most facilities. Sure, the facility probably can't do 5kw in every single cabinet, but I don't see how the 5-7kw in a single cabinet should really restrict things much.

    I guess when you're looking at $800 a month for 2x20A 120v you're looking at a different type of market than what will actually get you 208v at ~7kw that is being looked for.
    Karl Zimmerman - Founder & CEO of Steadfast
    VMware Virtual Data Center Platform

    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation

  19. #19
    Join Date
    Mar 2009
    Posts
    568
    Back a few months back when I did a tour of the datacenter I'm currently using, GoRACK, in Jacksonville FL - these guys had a nearly obscene amount of power available and great gear. I bet you anything those guys could make you a deal to bring your equipment in there with custom power arrangements to your requirements...

    --Chris
    The Object Zone - Your Windows Server Specialists for more than twenty years - http://www.object-zone.net/
    Services: Contract Server Management, Desktop Support Services, IT/VoIP Consulting, Cloud Migration, and Custom ASP.net and Mobile Application Development

  20. #20
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,710
    Quote Originally Posted by KarlZimmer View Post
    Interesting, pretty much the smallest 208v install we do is 30A 208v, which is 5kw, and the 2nd most popular is 2x 20A 208v, which is ~6.7kw. We can do that out of our oldest cage at 350 E Cermak which is ~10 years old at this point, so it sin't anything that new or interesting. Doing 5kw in a single cabinet should be EASY for most facilities. Sure, the facility probably can't do 5kw in every single cabinet, but I don't see how the 5-7kw in a single cabinet should really restrict things much.

    I guess when you're looking at $800 a month for 2x20A 120v you're looking at a different type of market than what will actually get you 208v at ~7kw that is being looked for.

    FWIW, we did not quote for any 208v power, so his remarks do not apply to us.
    We COULD provide 6.7KW in a cabinet, however it will not be at the price point he is trying to achieve.
    There are a lot of desperate companies in LA though (hence the quotes for $800 for a 40A rack - not one of our quotes), so maybe he can still find a 2x20A 208v rack on the cheap some place.
    I wouldn't rule out LA completely because of the above and since he isn't too concerned about heat, power redundancy, etc... then it may work out well for him to use one of those company's for as long as they are around.
    Maybe someone can get creative and give him straight grid power and a rack in front of a window to blow all his exhaust out of .
    EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
    Bandwidth Billing | Inventory & Asset Management | Server Control
    Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management

  21. #21
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,302
    Isn't power extremely expensive in CA? I know that I pay a butt load for power on the house I've got out there. .18993/kWH after all fees and taxes. Granted, these are consumer rates ...
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  22. #22
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,957
    Quote Originally Posted by CGotzmann View Post
    FWIW, we did not quote for any 208v power, so his remarks do not apply to us.
    We COULD provide 6.7KW in a cabinet, however it will not be at the price point he is trying to achieve.
    There are a lot of desperate companies in LA though (hence the quotes for $800 for a 40A rack - not one of our quotes), so maybe he can still find a 2x20A 208v rack on the cheap some place.
    I wouldn't rule out LA completely because of the above and since he isn't too concerned about heat, power redundancy, etc... then it may work out well for him to use one of those company's for as long as they are around.
    Maybe someone can get creative and give him straight grid power and a rack in front of a window to blow all his exhaust out of .
    Yeah, I think we're basically saying the same thing, that the issue is that he's looking for an $800 per 2x20A 120v cabinet type provider for ~7kw a cabinet in the first place. :-)
    Karl Zimmerman - Founder & CEO of Steadfast
    VMware Virtual Data Center Platform

    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation

  23. #23
    funkywizard: my hardware platform is already very well optimized. I use a 80+ Gold PSU (Corsair AX-1200) and measured 92% efficiency at 1100W with a clamp-meter. I don't want to go the trouble of running dual-redundant PSUs; it's not worth it (I would only gain max 2-3% in efficiency by running each PSU at half load). The servers are low-end single core CPU, 2GB RAM, diskless... basically optimized for cost. The GPUs, even though they are the "cheap" $500 gaming kind, represent 3/4rd of the cost of a server.

    I do think that $800 for 2x20A/120V is a great price, don't get me wrong guys. Also I would be ready to pay at least $1400/mo for 2x20A/208V. My remark about 208V was that I found no colo willing to sell a single 30A/208V circuit (as confirmed in this thread it is the smallest circuit that some offer, so others probably don't even sell it). I guess I will have to wait until I need 2x20A/208V to make the jump from 120 to 208.

    CGotzmann: you joke about blowing the exhaust out of a window but I would seriously love a colo using free air-cooling, exactly like Google, Microsoft, and Amazon do in their most advanced datacenters. That would attract customers like me who are in need of cheap power and don't mind higher temperatures (doesn't Google run some of their centers at 90 F?). Unfortunately the colo market is overly conservative and look with disdain at these ideas to cut cost, which is a mistake IMHO (untapped market).
    Last edited by xmrb; 02-01-2011 at 05:30 AM.

  24. #24
    Could your GPU focused servers deal with a higher intake temperature like you suggest?

    It's my understanding that most modern GPUs run close to heat thresholds during moderate usage in a "normal" environment, ie 70F ambient.
    ColoCrossing - Dedicated to Uptime
    Jon Biloh | jbiloh@colocrossing.com | 1.800.518.9716 | Skype: jbiloh
    DDOS Protected Colo & Dedicated Servers in Buffalo, Chicago, Dallas and Los Angeles

  25. #25
    Quote Originally Posted by xmrb View Post
    funkywizard: my hardware platform is already very well optimized. I use a 80+ Gold PSU (Corsair AX-1200) and measured 92% efficiency at 1100W with a clamp-meter. I don't want to go the trouble of running dual-redundant PSUs; it's not worth it (I would only gain max 2-3% in efficiency by running each PSU at half load). The servers are low-end single core CPU, 2GB RAM, diskless... basically optimized for cost. The GPUs, even though they are the "cheap" $500 gaming kind, represent 3/4rd of the cost of a server.

    I do think that $800 for 2x20A/120V is a great price, don't get me wrong guys. Also I would be ready to pay at least $1400/mo for 2x20A/208V. My remark about 208V was that I found no colo willing to sell a single 30A/208V circuit (as confirmed in this thread it is the smallest circuit that some offer, so others probably don't even sell it). I guess I will have to wait until I need 2x20A/208V to make the jump from 120 to 208.

    CGotzmann: you joke about blowing the exhaust out of a window but I would seriously love a colo using free air-cooling, exactly like Google, Microsoft, and Amazon do in their most advanced datacenters. That would attract customers like me who are in need of cheap power and don't mind higher temperatures (doesn't Google run some of their centers at 90 F?). Unfortunately the colo market is overly conservative and look with disdain at these ideas to cut cost, which is a mistake IMHO (untapped market).
    Ah, ok, yeah it does sound like a pretty efficient setup. Not sure what else you can do here other than make sure you're using a higher % of the power that a circuit allows.

    As to the blowing heat out a window, I don't think that's going to fly in Los Angeles during the summer
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

Page 1 of 2 12 LastLast

Similar Threads

  1. Usings GPU's in a server
    By ultimategp5 in forum Hosting Security and Technology
    Replies: 5
    Last Post: 02-22-2010, 11:07 AM
  2. Windows 7: Bsod with two GPU's
    By Bjørn-Erik Hansen in forum Computers and Peripherals
    Replies: 4
    Last Post: 08-01-2009, 05:39 AM
  3. Interest in GPU solutions?
    By hivelocitygm in forum Dedicated Server
    Replies: 10
    Last Post: 07-30-2009, 07:27 PM
  4. MediaTemple - 1000 GPU's isn't as much as it seems...
    By Sloth_Boy in forum Web Hosting
    Replies: 18
    Last Post: 12-08-2006, 10:15 AM
  5. Socket AM2 And GPU
    By dale1991 in forum Web Hosting Lounge
    Replies: 6
    Last Post: 11-22-2006, 08:23 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •