Comment by cobbzilla
8 hours ago
I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower.
I don’t know if that’s 100x more than I’d pay (opex-wise) with an nvidia setup, but I can say the one-time capex is a great deal cheaper. Avoiding VRAM and DDR5 (fast DDR4 should be OK) are the biggest cost savers. ECC RAM is worth the extra price. General datacenter-quality hardware has less price sensitivity, and plenty of bang for your buck.
Keep in mind that just because it has dual 750W power supplies that doesn't mean it's what its load will be, for a full CPU loaded wattage figure you'd need basically a pair of kill-a-watts plugged in inline on the feed for each poewr supply and then run stress-ng with artificial cpu stress on all cores for an hour.
Under heavy inference load you will find that the cpu usage is actually less as the bottleneck is the RAM bus speed. An older 2U rack server that is 600W load (typically a 1+1 power supply server when plugged into two kill-a-watt would show 300W on each, equal load balancing) when maxed out with stress-ng might be only 450W total running inference.
If you have 600kWh used in a month by running something 24x7 and your power is $0.15 a kWh, that's more like $90/mo (not counting cooling or any ancillary costs for the environment where it's in).
If you actually were running this thing at 80% or 100% load, then the first thing you'd want to is get a better PDU and then connect your servers to that (48V DC).
One of the problems in buying used/refurb x86-64 rack servers for test and development/proof of concept environment, is that by volume in the market, there's not that many -48VDC power supplies going around, because maybe 5-10% of enterprise customers buy them. Resulting in many fewer units ending up on the resale market.
The options for AC power supplies for servers with 2 or 4 load sharing redundant power supplies are a lot greater. If you were buying all new hardware and starting from a clean sheet of paper design with lots of money to spend, absolutely. At that point also start looking at higher voltage DC distribution stuff related to open compute platform and 800VDC.
But if I were trying to make the absolute most use of $20,000 to put together a 3TB RAM server (48 x 64GB DIMMs), it would likely end up AC powered.
(Context: Parent comment was edited after I wrote this comment)
Where in the world are you finding that much RAM in a racked server for $200/month?
I think he means electrical bill at his estimated wattage load of the server and his known kWh cost, not rented server/hosting cost.
Aha, right. That makes a lot more sense.
At my house. I have 5Gbps fiber and could pay for 10 or 25 if I need it.
Gotcha. But to be clear, you’re talking only about energy usage, correct?
1 reply →