Comment by pu_pe
15 hours ago
Inference will continue to be profitable while we don't have enough compute in the world to serve the demand we have. Also, I don't see why it's a problem that the end game for inference of open models will be relatively low, ie 5% or so. You can have a huge industry running on these margins. Though I would note that hyperscalers have been selling compute commodities with a high margin for a long time.
Demand is elastic, there's no fixed pool of it. What determines profitability isn't merely supply constraints but how much margin your competitors are willing to accept, more or less.
It's not a huge problem for users but if margins are low it's a huge problem for the debt funded datacenter buildout. It's just pure finance at that point, about whether those GPUs can generate enough yield to beat deprecation and financing costs.
The big clouds generally don't sell compute at high margins. The margins differ a lot between services. Services like core VMs and object storage are usually low margin or even break even. Where they make the money is things where they've got proprietary software with competitive advantage, like managed scalable forks of Postgres, or egress charges that are there to create lockin. Managed DBs are pure SaaS and can generate SaaS like margins.