Results 1 to 25 of 33
Thread: 7kW GPU farm
-
01-31-2011, 12:42 AM #1Newbie
- Join Date
- Jan 2011
- Posts
- 5
7kW GPU farm
I own a GPU Farm currently consuming up to 5 kW. It is made of 3U nodes consuming 1100W each (9.2A @ 120V) at full load and continuously. I need to expand to at least 7 kW soon and am looking for colo hosts in the Los Angeles area. I would appreciate some expert advice...
I have done some research, obtained some price estimates. First problem is that power in LA is expensive. The best offer I have is $800/mo for one cabinet with 2x20A.
Now get this: I can't even put 2 nodes on one circuit because that would be 18.4A, and continuous loads should not exceed 80% of the circuit rating, therefore should not exceed 16A on a 20A circuit. Which means the above offer only allows me to put 2 nodes per cab, or 1 per 20A circuit.
Because this is a ridiculous waste of power and space (only 18.4A used out of 40A, only 6U used out of 42U), I plan to downclock the GPUs to drop the power consumption of each node so I can put 4 per cab, or 2 per circuit. In fact this is what I am currently doing with some nodes.
But still I don't like wasting my hardware resources by downclocking. The best would be 208V/20A circuits: 3 nodes per circuit amounts to 15.9A... perfect. However I found no colo willing to sell 208V at my scale (5-7 kW).
Any suggestion?
Who sells 208V?
Another option would be a single 120V/50A circuit (4 nodes on it = 76%) but who sells 50A on a single circuit?
Also do you guys think I can find a better price than $800/mo for 2x20A in LA?
Thanks in advance.
-
01-31-2011, 01:30 AM #2Web Hosting Master
- Join Date
- Mar 2003
- Location
- chicago
- Posts
- 1,781
i dont think you will do better then $800 for 2 20a
-
01-31-2011, 02:05 AM #3Junior Guru Wannabe
- Join Date
- Jan 2011
- Posts
- 78
Contact Coresite - you'll save a boatload of money on power because they use utility billing, i.e. per kWh. Also they'll give you the power in whatever increment that they have breakers for. I don't think there's any such thing as a 50a breaker for a single phase, but surely 30a 208v to a single cabinet will fly. You might also be charged an extra fee of some kind for the "additional space" they'll allocate for your order. By that, I mean that you're using more than the typical 150 watts per sqft. density, so they're sure to charge you for "more space". Coresite has two LA facilities.
-
01-31-2011, 02:14 AM #4Web Hosting Master
- Join Date
- Aug 2007
- Location
- L.A., CA
- Posts
- 3,710
Due to paying for the extra space for the density, the cost will not be any different. Either way you're paying for the density. What matters is the total final cost, and all the power requires a lot of cooling.
EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
Bandwidth Billing | Inventory & Asset Management | Server Control
Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management
-
01-31-2011, 02:16 AM #5Web Hosting Evangelist
- Join Date
- Sep 2008
- Posts
- 515
What kind of facility do you need to be in? Unfortunately, for $800 a month in a city like LA, you won't get much in the way of redundancy...if you even need that. Especially at the power consumption you plan on having. Have you considered moving the colo out of LA?
█ GenesisAdaptive.com · IP Transit and Connectivity, Colocation, & Enterprise IT Services
█ Available in 9 Metros, 3 Continents, 1/10/40/100G Ports.
█ Follow us on Twitter @GenesisAdaptive!
█ Phone: (309) 807-5261 · Email: leec @genesisadaptive.com
-
01-31-2011, 02:20 AM #6Disabled
- Join Date
- Jun 2004
- Location
- Hyderabad India
- Posts
- 1,103
That's a lot of processing, look for any colo who say high density who provide more power per cabinet who will surely have 208v, if it's not bandwidth intensive, you can use UK/Asia providers where you can get 220V supply , some custom plans
Last edited by Krazy; 01-31-2011 at 02:23 AM. Reason: extra info
-
01-31-2011, 03:01 AM #7Newbie
- Join Date
- Jan 2011
- Posts
- 5
I have considered moving out of LA, especially because my workloads are not bandwidth intensive at all. I have looked at Portland, Oregon where power is inexpensive. But even there I have not been able to find 1 cab with 40A at less than $800. I would love to be proved wrong.
CGotzmann: I have no problem with paying for the extra space if this helps my host meet his power density target... you are right all that matters is the final cost.
ServerGurus: quite frankly I don't need much redundancy or anything... just power.
Thanks for the info guys (andrewinet, etc). I'll keep looking but so far this offer in LA looks like the best.
-
01-31-2011, 05:04 AM #8
I would find it hard to believe a single server pulls down 1100w unless it's got multiple graphics cards in it. You might try removing one card from a machine rather than downclocking them. Providers can also probably do 30a circuits for you for 50% more than the cost of a 20a circuit. You should be able to get two nodes onto a 30a circuit, which would be less wasteful than 1 per 20a.
Power in LA definitely is expensive. Utah has some of the lowest power costs in the country if that's at all an option for you, you might look into salt lake city. Phoenix also has pretty low power costs and is a solid Tier-2 city as far as the internet backbone goes.
As far as it goes, I think you'll have a hard time beating $800 for 2x20a in LA.IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
01-31-2011, 05:09 AM #9IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
01-31-2011, 05:13 AM #10Newbie
- Join Date
- Jan 2011
- Posts
- 5
Of course my nodes have multiple GPUs. The 1100W have been measured at the wall outlet. In fact, some nodes have less efficient PSUs and even hit 1250W.
Removing 1 or 2 GPUs from each node is an alternative to downclocking, which I am considering too...Last edited by xmrb; 01-31-2011 at 05:23 AM.
-
01-31-2011, 05:48 AM #11
One thing to definitely consider if you've not done so already is getting an 80+ gold rated power supply. When using that much power, it's pretty much mandatory.
Here's the newegg list of 1100+ watt power supplies that are 80+ gold (they're all under $300, and even though they're desktop supplies, I would expect that they should fit into a typical 3u rackmount case):
http://www.newegg.com/Product/Produc...1W%20-%201500W
Mind you, PSU's watt ratings are typically from delivered DC power, not wall power, so even if you're pulling 1100 watts from the outlet, a 1200 watt PSU should still have some room as the system is probably "only" drawing 1000 watts. On the other hand, PSU's are most efficient when you're using close to 50% of their rated capacity, so running bigger or redundant power supplies might be a good option here.
What are the specs on the actual server itself? Does the CPU run close to capacity during calculations, or just the GPU? These days a single socket system can easily draw 100 watts or less, so the power you waste using 3 servers with 2 cards each (for example), instead of 2 servers with 3 cards each, could be made up by the fact that the PSU in each of those servers will be operating close to it's optimal range, making either option likely to have the same power efficiency.
Are these the desktop series, ~$500 apiece kind of cards, or the higher end Tesla kind of models? If it's the higher end Tesla, most of the cost of the system should be tied up in the cards and not the servers, so again, going with more servers with fewer cards each shouldn't increase total costs that much, and will give you more flexibility to run closer to the power limits of the circuit.
Now that I think of it, a lot of rackmount cases these days are perfectly capable of dual redundant or even triple redundant power supplies. If you're pulling 1100 wall watts, you're probably delivering 900-1000 watts to your equipment. Running dual redundant 1k watt PSUs would allow each PSU to deliver 500 watts to your system, putting each psu at the magic 50% utilization level, where efficiency is highest, reducing your power use potentially by around 5%. The other advantage here, is that you can use one outlet on one PDU / Circuit, and the other outlet on a different PDU / Circuit. This way, you could put the equivalent load of 1.5 servers on one circuit, by connecting both plugs on one server to a circuit, both plugs on a different server to a different circuit, and 1 plug each on a third server to each of the two circuits. Getting 3 servers onto $800 / mo sounds a lot better than fitting 2 servers onto $800 / mo.
The obvious downside is, if one of the circuits trips or loses power, the system that's "dual redundant" on both circuits will put it's full load onto the other circuit, causing it to trip the breaker. So it's not the best solution in terms of redundancy, but if you're just looking to get done with X amount of work for the lowest cost, it's the best way to go in your power profile. At 9.2a for one server, 1.5 servers would be like 13.8a, well within the 16a acceptable continuous usage profile, with a little left over for switches or whatever.
Of course, if power really is the biggest concern, then Utah, where industrial power rates are less than half of those in Los Angeles, is a good option. Or somewhere where you can use free cooling instead of air conditioning most of the year.IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
01-31-2011, 10:18 AM #12Master of the Truth
- Join Date
- Mar 2006
- Location
- Reston, VA
- Posts
- 3,131
Yellow Fiber Networks
http://www.yellowfiber.net : Managed Solutions - Colocation - Network Services IPv4/IPv6
Ashburn/Denver/NYC/Dallas/Chicago Markets Served zak@yellowfiber.net
-
01-31-2011, 12:08 PM #13Web Hosting Master
- Join Date
- Jan 2011
- Location
- UK
- Posts
- 776
-
01-31-2011, 12:13 PM #14THE Web Hosting Master
- Join Date
- Jan 2003
- Location
- Chicago, IL
- Posts
- 6,957
You're having trouble finding 208v, where are you looking? 208v is pretty standard from everything I've seen. Most of our new cabinet installs have been 208v, though we aren't in LA. If your facility can't/won't offer 208v options they must have NO control over their infrastructure or don't know how to build out a facility, so look elsewhere. There MUST be someplace in the LA area that can do 208v, and as you said, 20A 208v seems to make the most sense for you.
I would first make absolute certain though that you're using the most efficient power supplies possible, as you've said yourself, that can make a major difference, and now ~90% efficient power supplies aren't that expensive either.Karl Zimmerman - Founder & CEO of Steadfast
VMware Virtual Data Center Platform
karl @ steadfast.net - Sales/Support: 312-602-2689
Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation
-
01-31-2011, 01:50 PM #15Junior Guru
- Join Date
- Sep 2010
- Location
- Dallas, TX
- Posts
- 234
I am surprised that 208v is that hard to source in LA as well. Either your sales rep isn't asking the right questions to operations or you are looking at some very old data centers that lack the density to support it. At the very least, they should offer you the power in a cage environment (to help spread the power/sqft).
If you do go to 208v, make sure your auxiliary devices (if you have any) are capable of 208v as well. That's often overlooked until it's too late.
-
01-31-2011, 02:23 PM #16Web Hosting Master
- Join Date
- Jun 2001
- Location
- Denver, CO
- Posts
- 3,302
Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
Current specials here. Check them out.
-
01-31-2011, 02:48 PM #17Web Hosting Master
- Join Date
- Aug 2007
- Location
- L.A., CA
- Posts
- 3,710
He said that he can't find a colo that will do 208v at his density 5kw-7kw, not that 208v was not available.
EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
Bandwidth Billing | Inventory & Asset Management | Server Control
Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management
-
01-31-2011, 02:54 PM #18THE Web Hosting Master
- Join Date
- Jan 2003
- Location
- Chicago, IL
- Posts
- 6,957
Interesting, pretty much the smallest 208v install we do is 30A 208v, which is 5kw, and the 2nd most popular is 2x 20A 208v, which is ~6.7kw. We can do that out of our oldest cage at 350 E Cermak which is ~10 years old at this point, so it sin't anything that new or interesting. Doing 5kw in a single cabinet should be EASY for most facilities. Sure, the facility probably can't do 5kw in every single cabinet, but I don't see how the 5-7kw in a single cabinet should really restrict things much.
I guess when you're looking at $800 a month for 2x20A 120v you're looking at a different type of market than what will actually get you 208v at ~7kw that is being looked for.Karl Zimmerman - Founder & CEO of Steadfast
VMware Virtual Data Center Platform
karl @ steadfast.net - Sales/Support: 312-602-2689
Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation
-
01-31-2011, 03:07 PM #19Web Hosting Master
- Join Date
- Mar 2009
- Posts
- 568
Back a few months back when I did a tour of the datacenter I'm currently using, GoRACK, in Jacksonville FL - these guys had a nearly obscene amount of power available and great gear. I bet you anything those guys could make you a deal to bring your equipment in there with custom power arrangements to your requirements...
--ChrisThe Object Zone - Your Windows Server Specialists for more than twenty years - http://www.object-zone.net/
Services: Contract Server Management, Desktop Support Services, IT/VoIP Consulting, Cloud Migration, and Custom ASP.net and Mobile Application Development
-
01-31-2011, 04:04 PM #20Web Hosting Master
- Join Date
- Aug 2007
- Location
- L.A., CA
- Posts
- 3,710
FWIW, we did not quote for any 208v power, so his remarks do not apply to us.
We COULD provide 6.7KW in a cabinet, however it will not be at the price point he is trying to achieve.
There are a lot of desperate companies in LA though (hence the quotes for $800 for a 40A rack - not one of our quotes), so maybe he can still find a 2x20A 208v rack on the cheap some place.
I wouldn't rule out LA completely because of the above and since he isn't too concerned about heat, power redundancy, etc... then it may work out well for him to use one of those company's for as long as they are around.
Maybe someone can get creative and give him straight grid power and a rack in front of a window to blow all his exhaust out of .EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
Bandwidth Billing | Inventory & Asset Management | Server Control
Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management
-
01-31-2011, 04:18 PM #21Web Hosting Master
- Join Date
- Jun 2001
- Location
- Denver, CO
- Posts
- 3,302
Isn't power extremely expensive in CA? I know that I pay a butt load for power on the house I've got out there. .18993/kWH after all fees and taxes. Granted, these are consumer rates ...
Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
Current specials here. Check them out.
-
01-31-2011, 05:15 PM #22THE Web Hosting Master
- Join Date
- Jan 2003
- Location
- Chicago, IL
- Posts
- 6,957
Karl Zimmerman - Founder & CEO of Steadfast
VMware Virtual Data Center Platform
karl @ steadfast.net - Sales/Support: 312-602-2689
Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation
-
02-01-2011, 05:26 AM #23Newbie
- Join Date
- Jan 2011
- Posts
- 5
funkywizard: my hardware platform is already very well optimized. I use a 80+ Gold PSU (Corsair AX-1200) and measured 92% efficiency at 1100W with a clamp-meter. I don't want to go the trouble of running dual-redundant PSUs; it's not worth it (I would only gain max 2-3% in efficiency by running each PSU at half load). The servers are low-end single core CPU, 2GB RAM, diskless... basically optimized for cost. The GPUs, even though they are the "cheap" $500 gaming kind, represent 3/4rd of the cost of a server.
I do think that $800 for 2x20A/120V is a great price, don't get me wrong guys. Also I would be ready to pay at least $1400/mo for 2x20A/208V. My remark about 208V was that I found no colo willing to sell a single 30A/208V circuit (as confirmed in this thread it is the smallest circuit that some offer, so others probably don't even sell it). I guess I will have to wait until I need 2x20A/208V to make the jump from 120 to 208.
CGotzmann: you joke about blowing the exhaust out of a window but I would seriously love a colo using free air-cooling, exactly like Google, Microsoft, and Amazon do in their most advanced datacenters. That would attract customers like me who are in need of cheap power and don't mind higher temperatures (doesn't Google run some of their centers at 90 F?). Unfortunately the colo market is overly conservative and look with disdain at these ideas to cut cost, which is a mistake IMHO (untapped market).Last edited by xmrb; 02-01-2011 at 05:30 AM.
-
02-01-2011, 05:45 AM #24Web Hosting Master
- Join Date
- Oct 2004
- Posts
- 901
Could your GPU focused servers deal with a higher intake temperature like you suggest?
It's my understanding that most modern GPUs run close to heat thresholds during moderate usage in a "normal" environment, ie 70F ambient.ColoCrossing - Dedicated to Uptime
█ Jon Biloh | jbiloh@colocrossing.com | 1.800.518.9716 | Skype: jbiloh
█ DDOS Protected Colo & Dedicated Servers in Buffalo, Chicago, Dallas and Los Angeles
-
02-01-2011, 06:33 AM #25
Ah, ok, yeah it does sound like a pretty efficient setup. Not sure what else you can do here other than make sure you're using a higher % of the power that a circuit allows.
As to the blowing heat out a window, I don't think that's going to fly in Los Angeles during the summerIOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
Similar Threads
-
Usings GPU's in a server
By ultimategp5 in forum Hosting Security and TechnologyReplies: 5Last Post: 02-22-2010, 11:07 AM -
Windows 7: Bsod with two GPU's
By Bjørn-Erik Hansen in forum Computers and PeripheralsReplies: 4Last Post: 08-01-2009, 05:39 AM -
Interest in GPU solutions?
By hivelocitygm in forum Dedicated ServerReplies: 10Last Post: 07-30-2009, 07:27 PM -
MediaTemple - 1000 GPU's isn't as much as it seems...
By Sloth_Boy in forum Web HostingReplies: 18Last Post: 12-08-2006, 10:15 AM -
Socket AM2 And GPU
By dale1991 in forum Web Hosting LoungeReplies: 6Last Post: 11-22-2006, 08:23 AM