Comment by dmtroyer
17 hours ago
> Amortized cost of owning a DC full of GPUs is far less than $38/card/hr
what do you think it is, out of curiosity?
17 hours ago
> Amortized cost of owning a DC full of GPUs is far less than $38/card/hr
what do you think it is, out of curiosity?
Other replies covered this, but I was basing that comment on the prices that commodity providers charge for GPU-accelerated VMs. E.g. the going rate for a NVIDIA B300 (288GB RAM) and the VM around it is somewhere between $5 and $20 an hour [1].
These providers have very little incentive to charge unprofitable prices (except maybe capturing more of the market, but there's not much lock-in here), so it's reasonable to assume somewhere in that spread is the "true" cost of being in this business, all expenses included.
[1] I'm using https://gpus.io as a reference here, and that's on-demand prices, which will tend to be the highest relative to other options.
(tongue mostly in cheek) sounds like I should be renting out these platforms and selling inference on them to OpenRouter
You can pay off a high end Nvidia DGX server at just over $7/hr/card over a 1 year period.
$38/card/hr is a ton of money. That's $2.6 million per 8-card server per year.
The cost of power/physical space/interconnect by comparison here is negligible.
...well you can get B200 in cloud for ~$6-14/hour and that's after all hosting costs and profits company make of it soooo yeah