Comment by freakynit
18 hours ago
For the ones who are still wondering how so much profits, inference is a highly profitable domain: https://inferencex.semianalysis.com/profit-estimator
18 hours ago
For the ones who are still wondering how so much profits, inference is a highly profitable domain: https://inferencex.semianalysis.com/profit-estimator
I think we need to collectively get a grip on statements like this.
There is no reason to think inference can or will remain interestingly profitable even if you assume pure-play inference providers who don't do any model training. Serving an LLM isn't child's play but it's also not so hard as to count as a moat. Inferencing is:
1. Selling a largely undifferentiated product.
2. Into a highly competitive market.
3. To customers who are primarily sensitive to price.
Industries like this have always existed and in the modern era tend to have margins of about 3% to 5%. Any level of margin higher than that gets competed into the ground.
If you want to claim inference is highly profitable in general, you're going to have to answer why it won't end up like supermarkets or airlines, both of which have bigger moats than inference.
The other problem with this analysis is obviously that you can't just stop training models. Even if you don't care about improving capability or performance, the knowledge in their weights gets stale. Cost of inferencing has to take into account the ongoing cost of training, you can't cleanly split one from the other.
Amen. AI boosters look at currently high profit margins on inference and blindly accept claims that these inference profit margins will continue. Of course those margins depend on historically massive R&D spend on training. The moment that training stops, AI returns to a commodity market and the inference profits end. Big labs are trying as hard as possible to keep the gravy train running but they can all see it won't last forever, hence the rush for IPO & regulating away open models.
Boosters are right that AI is a historically impactful product, but big labs will be unable to meaningfully profit from ushering in the AI era just like Netscape failed despite ushering in the internet era.
"If you want to claim inference is highly profitable in general, you're going to have to answer why it won't end up like supermarkets or airlines, both of which have bigger moats than inference."
- I don't... "inference is profitable" is as of now. No one knows future, neither have I commented anything on it.
The statement is valid as it stands.
Inference will continue to be profitable while we don't have enough compute in the world to serve the demand we have. Also, I don't see why it's a problem that the end game for inference of open models will be relatively low, ie 5% or so. You can have a huge industry running on these margins. Though I would note that hyperscalers have been selling compute commodities with a high margin for a long time.
Demand is elastic, there's no fixed pool of it. What determines profitability isn't merely supply constraints but how much margin your competitors are willing to accept, more or less.
It's not a huge problem for users but if margins are low it's a huge problem for the debt funded datacenter buildout. It's just pure finance at that point, about whether those GPUs can generate enough yield to beat deprecation and financing costs.
The big clouds generally don't sell compute at high margins. The margins differ a lot between services. Services like core VMs and object storage are usually low margin or even break even. Where they make the money is things where they've got proprietary software with competitive advantage, like managed scalable forks of Postgres, or egress charges that are there to create lockin. Managed DBs are pure SaaS and can generate SaaS like margins.
Exactly, inference is like groceries or electricity.
Because LLMs are like evolutionary crabs, they all get to the same point.
I think hardware scarcity plays a part in inference margins. They would get squeezed super quickly if anyone who wanted could just build an inference cluster, but that's not currently the case.
I would assume hardware production has been ramping up significantly and it's more of a temporary supply chain issue though. While it could last multiple years, that's certainly not long enough for the hyperscalers to get a return on investment. Obviously more complicated than this, but still... I'm not buying.
We're also starting to see efficiency gains, so we might meet halfway.
Maybe but there are huge blocks of capacity about to be energized or handed over and the payment clock starts ticking immediately.
Complete and accurate, thanks.
Accounting for initial acquisition cost and typical depreciation/failure rate? I assume they already consider colo price, electricity, etc
70%+ net profits margins are for rented infra, that means no major upfont expense. Also, these margins are against 60% utilization only.
You can well imagine if you own your infra and can make utilization over 90%, where these go.
And then, my link above is for K3 model, which, compared to ds flash models, is highly unoptimized.
If I had a few million lying around, I would have invested in inference.
I would be surprised if there was such a glut of undifferentiated inference providers for open models otherwise[0]. Possibly just people trying to minimize the loss from already-committed capital, but I don't see AWS and DO selling inference via OpenRouter unless they're making a decent profit from it.
0: https://openrouter.ai/openai/gpt-oss-120b#providers
Fascinating site. Thanks for linking. However note if you change to "open router" pricing in some cases even B200 which is ~2y old has negative margin (and AFAIK ppl are using longer amortization than 2y - I do not know how cost are calculated here though).
1. The link is for rented infra. That means the inference provider doesnt have to worry about deprecation.
2. Newer gen nvidia gpu's like GB300 are way more profitable.
Why isn't ordered by profit % per GPU? It' surely the most important indicator