Comment by nostrademons
4 hours ago
These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost.
Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.
You have to power it all the time, but the amount of power it uses while it’s on will change by up to a few orders of magnitude depending on the gpu. It’s not uncommon for a gpu to be pulling just a couple of watts at idles and several hundred at full tilt.
So the only way it’s a fixed cost is if you don’t pay for power. If you only consider the cost of the power, it might still be cheaper paying for an api.
Good points. Speaking from experience, it’s really hard to make it more economical than using a cloud AI service - even if you utilize the GPU to its fullest. There are, of course, other benefits, such as privacy/control/compliance/security, which should be the real reasons to do this, not cost.
You sweet sweet summer child.
Spoken like a guy who has never had to maintain bare metal infrastructure ops at scale. These things break, need re-imaged, have parts that break, have to be configured (now you need provisioning pipelines, monitoring, alarms, etc), have to be maintained when something goes wrong (swapping hardware and software, debugging the alarms into actually figuring out which bits are broken and/or misconfigured), have to be catalogued, have to be planned for, deprecated, and the finances accounted for through complicated accounting to show investors the Capex at quarterly meetings.
Then you have to make fleet decisions on how much of each type of server you will want to buy, expanded storage, how long you will support each generation of server, when you will order new hardware, how to order new hardware, lag for real world installs, hiring actual humans to fly around to all of your datacenters and do the actual installations / maintainance / etc. Then you will have to do contracts with individual datacenter operators for margins, electricity rates, hosting contracts, white glove ops hourly rates, etc.
Businesses that own their own hardware tend to have a lot of employees whose jobs are maintaining it and running the business side of that.
You do realize that a fully utilised GPU uses about 10 to 50 times more power than an idling one?