Comment by HarHarVeryFunny

That turns out to only be a small part of the cost.

Say a $30,000 GPU has an operational lifetime of 3 years, and draws 1000W. If it is running 24x7 then over those 3 years it will consume 26,280 kWh of electricity, at a cost of say $0.10 per kWh, giving a lifetime electricity cost of $2600, or less than 10% of the cost of the GPU itself.

So, to a first approximation the cost of serving a model is the amortized GPU cost.