Comment by jeremyjh
19 hours ago
Datacenters full of GPU hosts aren't like dark fiber - they require massive ongoing expense, so the unit economics have to work really well. It is entirely possible that some overbuilt capacity will be left idle until it is obsolete.
The ongoing costs are mostly power, and aren't that massive compared to the investment.
No one is leaving an H100 cluster not running because the power costs too much - this is why remnants markets like Vast.ai exist.
They absolutely will leave them idle if the market is so saturated that no one will pay enough for tokens to cover power and other operational costs. Demand is elastic but will not stretch forever. The build out assumes new applications with ROI will be found, and I'm sure they will be, but those will just drive more investment. A massive over build is inevitable.
Of course!
But the operational costs are much lower than some people in this thread seem to think.
You can find a safe margin for the price by looking at aggregators.
https://gpus.io/gpus/h100 is showing $1.83/hour lowest price, around $2.85 average.
That easily pays running costs - a H100 server with cooling etc is around $0.10/hour to keep running
And a massive overbuild pushes prices down not up!