Comment by schneehertz

There has always been a lot of evidence that prices are only going down. The fact that you could already get DeepSeek or GLM inference from 20+ providers on openrouter.ai was already sufficient evidence that inference was profitable and being commoditized.

People that want to believe something different as a coping mechanism will not be swayed by this evidence either.

> I think everyone has known that inference is profitable overall for the major providers

> The major providers are unprofitable after you add in the training and research costs and other overheads

It's meaningless to say that selling something is profitable if you are ignoring the cost to produce it.

The profitabiity (or not) of AI models needs to be calculated on a per model lifetime basis - what did the model cost to develop and train, what is the lifetime revenue the model generates, and what is the lifetime operational cost of the model (including amortized GPU cost), etc.

It makes no more sense to calculate "inference" profitability when ignoring training cost than it would to calculate it when ignoring GPU cost.

> The major providers are unprofitable after you add in the training and research costs and other overheads

The very short lives of all these models also factor in.

I think the funny thing is the more inference, the more tokens a specific version of a model spits out, actually does make it more profitable.

But somehow a lot of people truly think that each token generated increases the cost of OpenAI/Anthropic/... and shout bloody murder about "subsidies"

And I'm certain there are people out there that actually think the API prices of OpenAI and Anthropic somehow equal their electricity costs.

>I think everyone has known that inference is profitable overall for the major providers for a while

Have you seen any place that is not as tech-savy as HN? e.g. some Reddit subreddits?

Barely anyone is aware other than people who are already somewhat deep into this whole topic.

Yes. I've always suspected inference is profitable.

But I'm also interested is retraining needed? how relevant will a model be in N years if it's just "static", and if you do need to retrain and fine-tune over time, is it still profitable? how often is that needed?

For most cases it's probably ok, often times answers are general or basically a web search anyways, but if my Agent started blasting outdated syntax it would suddenly be way worse for coding for example.

> I think everyone has known that inference is profitable overall for the major providers for a while

A lot of AI haters like Ed Zitron and their readers have refused to update on this. I don't know if any of them have yet, but it'll take a year or two for the idea to fully filter out of their podcasts anyway.

It’s not training and research that kills you here, it’s your own investors ROI expectations that murder you cold when faced with Open Source commoditizing you, from a country that spent 5% of the sum you spent to get to the same position and then gives the recipe away to everyone.

> I think everyone has known that inference is profitable overall for the major providers for a while

Unfortunately while in the tech industry some might know that, the general public are still being misled (especially by certain folks who’ve made a living off downplaying AI) to believe inference itself is unprofitable.

> The major providers are unprofitable after you add in the training and research costs and other overheads.

This is correct for now, but I recently read that 90% of Anthropic's code is written by Claude Code itself. So, if an AI is using its previous version to train the next version, we're not far from the day where these models will take up the decision making process too. For eg: Decisions API by OpenAI or Jev is based exactly on this. What if a model like Jev sits behind Opus 5.5 and guides it to take the next step? Wooh.

Probabilistic predictions right now are at 99% for frontier models. They will never be 100%, but if humans rate themselves 99.9% in decision making, that benchmark can be broken.

Yeah I think one of the less considered costs when people talk about this is the human cost of training. We all talk about the billions spent on compute, but I don't think anyone's got good numbers on how much RLHF is costing but it's got to be pretty large.

As someone who has long bought into the idea that the labs are not profitable on inference, I have a question. How is it simultaneously true that there is more demand for inference than supply (apparently by orders of magnitude), but also that inference is so cheap that labs can make healthy margins? It seems like the inference providers, not the model providers, should have the leverage. Why haven’t they said, “give me a bigger cut of each token or I won’t serve your model”?

Also, what happens when the inference providers suffocate on their own debt burdens? Who will serve inference?

The only moat is in datacenters. The software has no moat, since 1) it's already trained on stolen data, and 2) someone with access to AI will be able to reverse engineer a proprietary implementation eventually, the same AI is being used for developing AI.

That's why so many talks about regulations. Regulations means: ban outside competition, just like US bans Chinese EVs today.

I would classify training and research more as capital investment than overhead.

Most people in my experience refuse to believe it for some reason.

Well, as that means there's lots of money to be made lowering costs for research and training! What tools have we used in the past to lower those costs? Automation.. Hm, what if we had AI completely automate AI research? Uh oh, we just rationally found outselves in a trajectory where we lose control of our own trajectory because of economic incentives.

It is wild how fast we're hurtling towards this outcome, lmao.

I doubt that, I got downvoted several times pointing out inferencing is insanely profitable.

Even take account of R&D cost it is still profitable, because you do not actual retrain the base model every month, only distill and post train from the same foundation.

heck, I think even including the giant cost of the top tier researcher salaries, they are still profitable.

then, how are they still losing money? just read what they are actually spend the most budget: always the hardware. those companies will never profitable, because they will never stop spending money to expand their computational prowess.

If the frontier labs are struggling under the training costs and the leading open source models are making their progress by distilling from the frontier labs it sounds a bit like there’s something wrong with the economics here or that progress is going to slow quite a bit without the amount of investment.

I don't think so. The number of comments I see from people talking about AI, there are plenty who still think the price is going to shoot up. Some even talk this way about models you can run locally on high end yet still consumer level hardware.

I think the more someone works with AI the more likely they realize that inference does not have to be sold at a loss, but many people's AI knowledge seems quite dated and it sounds almost like it becomes a convenient conspiracy theory for some, a way to convince themselves that the market changes that are already happening a temporary (and I don't mean in a Jevon's paradox sort of way, but that companies will go away from AI).

I suspect a lot of this is fueled by a few stories of companies who went all in on spending as much as possible on AI and then begun to walk that back, without realizing that was more about bad incentives and over promises of what AI could do (especially 1/2 model generations ago). Some also confuse running the SotA models at the fullest capacity costing more and more with a given problem costing more and more to solve, not realizing that the cost per problems is trending down but newer and larger problems can be solved for even more, so the costs at the top are rising despite falling prices. Also things like the subscriptions getting more expensive, though how much of that is because developers are doing far more than they were 6 or 12 months ago (and in some cases there was subsidization that is going away). And maybe a bit with the free models starting to not be upgraded or the harnesses getting worse performance of lowered free limits.

But you are mixing up different categories.

Most of your questions are things that are relevant to everybody due to a possible economic crash. And you are right to point this out.

But for me as an AI user those aspects are not really relevant in my decisions. If an AI bubble bursts providers will still want to sell me those tokens at that price and yes development will slow down for a time which does change not a lot for me.

Inference is very profitible when you can deal with the batch sizes Anthropic, Deepseek and OpenAI are dealing with, renting GPUs for inference is even more profitible.

The US is dealing with different tax code where buying GPUs function as as great way to offset taxes, and having boatloads of debt is rewarded in many cases. I think there's lots more (crooked) reasons why inference is even more profitible in the US.

100% the right questions. The implied thought process of so many on this thread: Inference profitable >> Frontier Labs' are profitable overall. Sigh...

I seem to recall the leaked Liang Wenfeng speech gave a figure of 8-12(?) months of inference recouping the training costs of the model.

What about sourcing the asset itself? It's like a factory is it not?

You pay for the asset (tenancy in a data center) and then you pay for housing, powering, cooling, etc.

I think it’s pretty clear from the cherry picked numbers at Anthropic.

Demand is the issue. Demand only spikes with a sota model release then in drops back down to unprofitable levels when factoring in training a new model and build out.

So your stuck in an unprofitable cycle of losing revenue when your not releasing sotas but losing profit when you are.

This goes back to googles original memo “there is no moat”.

Where in you can’t stop training sotas because you’ll lose customers to the next inference model above you instantly.

Google has seemingly defeated this by finding models cheap and good enough for their existing platforms resulting in a moat of distribution. Something OpenAI and Anthropic seems to lack.

Unless the money of the investors is going somewhere else...

Presumably cards run on electricity?

Nothing but hardware access, which is getting harder for the average person but for a giant corporation I see no reason they can’t compete now at the local model level

> My workflow is one where I am heavily involved in the process and I don't like giving the agent too much autonomy

For such workflows it barely matters which of a dozen models gets used.

Whereas if you're handing off all dev and no longer reading code, just making CTO-level architecture decisions, then there are only 2~3 models right now worth using, none cheap.

> I'm daily driving DeepSeek 4.1 flash (at API prices) and every now and then I give Opus 5.5 a try and don't notice anything missing for me.

I was doing the same (along with GLM 5.3 Flash) for months, and then gave Claude Sonnet 5.5 at medium effort a try. The difference was like night and day. Where both DeepSeek 4.1 Flash and GLM 5.3 Flash kept making mistakes and would have to do a bunch of iterations to find and fix them, Claude Sonnet 5.5 just did it right on the first try.

In my experience, while DeepSeek 4.1 Flash and GLM 5.3 Flash are a lot cheaper per token, they're just as or maybe even more expensive when you factor in all the extra tokens that are need to troubleshoot and fix their mistakes -- and even more expensive if I factor in my own time wasted doing the same, not to mention the frustration of having to deal with that.

DeepSeek 4.1 Flash also hallucinates a lot.[1]

I'm hoping all this improves with time, and the gap between the competent frontier models and open weight models narrows.

[1] - https://www.youtube.com/watch?v=P4dTq4X8bqk

That’s the Model Welfare speaking, not your bias.

OpenAI's entire business is trivially profitable when you attach an ad network to ChatGPT.

Quick, someone point out the one billion user system with intent + personal data rich that doesn't result in a $100+ billion high margin ad network.

It is that simple and absolutely nothing can stop the gigantic ad networks that will be built up around personal agents.

[dead]

Huge, generational mistake for the USA to be leaning so hard into fossil fuels right now and actively trying to prevent buildout of solar and wind power. That decision is just going to hurt more and more in the coming years as China's electricity costs continue to plummet while the US' continues to depend on unsustainable fossil fuel infrastructure.

China is investing in cheap and secure energy, which includes renewables, but it also include coal: https://www.reuters.com/business/energy/chinas-coal-power-ri...

The latter is great as a way of getting an edge in business and reducing reliance on oil supplies, but its also environmentally highly damaging.

> You missed the bit where they used the word "revenue".

No they didn't...

> I suspect if you factor in all the various overheads of running datacentres the actual value is lower.

That's literally exactly what the quoted part addresses: "Amortized cost of owning a DC full of GPUs is far less than $38/card/hr". Cost of running GPUs is far less than $38/hour, revenue is $38/hour, ??? , profit

Other replies covered this, but I was basing that comment on the prices that commodity providers charge for GPU-accelerated VMs. E.g. the going rate for a NVIDIA B300 (288GB RAM) and the VM around it is somewhere between $5 and $20 an hour [1].

These providers have very little incentive to charge unprofitable prices (except maybe capturing more of the market, but there's not much lock-in here), so it's reasonable to assume somewhere in that spread is the "true" cost of being in this business, all expenses included.

[1] I'm using https://gpus.io as a reference here, and that's on-demand prices, which will tend to be the highest relative to other options.

You can pay off a high end Nvidia DGX server at just over $7/hr/card over a 1 year period.

$38/card/hr is a ton of money. That's $2.6 million per 8-card server per year.

The cost of power/physical space/interconnect by comparison here is negligible.

...well you can get B200 in cloud for ~$6-14/hour and that's after all hosting costs and profits company make of it soooo yeah

https://inferencex.semianalysis.com/profit-estimator

When that happens, the chips stocks will have tanked back to sane levels and the doom posting will be about that. That will put us in the 2001 era of post dot com bubble, and the following few years will show us who the winner are

And it seems we're getting just 10x more software nobody uses.

Why do you posit that this is a lot more important than the Internet?

> AI as a business will be 10x larger than it is today within the next 20 years.

I would argue it will be 10x in 2-3 years the way things are going.

> That is the danger of open source models. They destroy monopolies and shrink margins.

Danger? You mean benefit?

> That is the danger of open source models. They destroy monopolies and shrink margins.

It's unclear that the leading open-source models could exist without having distilled the frontier models. If that's true, the "danger" is that these upstream models don't get built any more (because of the tremendous cost), and we end up just optimizing what we have. Also unclear whether this would be a net win or loss for humanity.

It is mostly response to claims of "all LLM are subsidized and blatant bubble, LLM are useless" etc

China generates more electricity from coal than from all renewables put together (49.7% per https://oilprice.com/Latest-Energy-News/World-News/Solar-Ove...).

They have more installed renewable capacity than coal, but renewables have low capacity factors, solar is less than 50% by definition. The renewable share is growing fairly rapidly, but it is by now means almost entirely renewable.

Its not 10 months worth of revenue should be equal to purchase price. He is saying 10 months margin will cover the hardware cost, so revenue is ofcourse going to be higher, quote below.

"This is our API pricing. For our API pricing, we consider a reasonable profit to be roughly: we buy a batch of equipment from the market, and recoup the cost in ten months. I think that's a reasonable profit."

Isn't electricity a pretty major factor even over such a short fraction of the hardware's lifetime?

No, those $3 are on the high end. If you do not factor in hardware depreciation, at China's low electricity prices (largest variable cost and China has a surplus larger than many country's entire consumption) and with GB200 which is one of DeepSeek's main platforms, the operation cost is actually less than 50 cents per hour most likely. If you factor in hardware depreciation this rises significantly but almost certainly stays under 3 usd per hour, probably closer to 2.

Orinally it was an educated guess based on rental prices I've seen but Opus 5.5 and Gemini 3.8 Flash both arive at the same numbers based on DeepSeek's Zhihu post and GB200 specs. Around 30 cents before depreciation and $2 to 3.50 including it.

Serving LLM inference is more efficient with more concurrent requests. For a single inference request, the bottleneck is GPU memory bandwidth, as the inference engine spends most of its time waiting to move values in memory. With concurrent requests, it's possible to service multiple requests in a single pass through the model weights, meaning you can utilize more of the GPU/APU's compute.

So you have to price your local hardware against however many subscriptions you would need to run a dozen concurrent agents around the clock.

But even then, I would guess subscriptions are priced on the basis that they don't expect every user to utilize their entire usage limit. I.e. power users are subsidized by casual users.

All of it is really about scale, same reason as a single person you can't sell french fries as cheap as McDonalds.

Because the datacenter setups serve multiple users concurrently while your mac mini only serves one. You would also lose money hand over fist renting/buying 4xH200s just to serve deepseek flash to yourself.

These datacenters run the GPU's close to 100% usage 24/7. The Mac mini at home probably has <25% utilization. And then there's economies of scale, wholesale discounts, etc.

The Mac mini GPU also isn't as optimized as the datacenter GPUs

You can batch calls if you're serving a lot of people at once.

[flagged]

I just installed a $16k GPU at home - super pricy to be sure, but when you price it against hiring 4 junior developers working around the clock it's a steal

Amen. AI boosters look at currently high profit margins on inference and blindly accept claims that these inference profit margins will continue. Of course those margins depend on historically massive R&D spend on training. The moment that training stops, AI returns to a commodity market and the inference profits end. Big labs are trying as hard as possible to keep the gravy train running but they can all see it won't last forever, hence the rush for IPO & regulating away open models.

Boosters are right that AI is a historically impactful product, but big labs will be unable to meaningfully profit from ushering in the AI era just like Netscape failed despite ushering in the internet era.

"If you want to claim inference is highly profitable in general, you're going to have to answer why it won't end up like supermarkets or airlines, both of which have bigger moats than inference."

- I don't... "inference is profitable" is as of now. No one knows future, neither have I commented anything on it.

The statement is valid as it stands.

Inference will continue to be profitable while we don't have enough compute in the world to serve the demand we have. Also, I don't see why it's a problem that the end game for inference of open models will be relatively low, ie 5% or so. You can have a huge industry running on these margins. Though I would note that hyperscalers have been selling compute commodities with a high margin for a long time.

Exactly, inference is like groceries or electricity.

Because LLMs are like evolutionary crabs, they all get to the same point.

I think hardware scarcity plays a part in inference margins. They would get squeezed super quickly if anyone who wanted could just build an inference cluster, but that's not currently the case.

Complete and accurate, thanks.

70%+ net profits margins are for rented infra, that means no major upfont expense. Also, these margins are against 60% utilization only.

You can well imagine if you own your infra and can make utilization over 90%, where these go.

And then, my link above is for K3 model, which, compared to ds flash models, is highly unoptimized.

If I had a few million lying around, I would have invested in inference.

I would be surprised if there was such a glut of undifferentiated inference providers for open models otherwise[0]. Possibly just people trying to minimize the loss from already-committed capital, but I don't see AWS and DO selling inference via OpenRouter unless they're making a decent profit from it.

0: https://openrouter.ai/openai/gpt-oss-120b#providers

1. The link is for rented infra. That means the inference provider doesnt have to worry about deprecation.

2. Newer gen nvidia gpu's like GB300 are way more profitable.

The top quality frontier models are used to compress / improve the earlier, lesser models, to make them more efficient. So that pushes costs way down on previous models.

Isn't OpenRouter only part of the traffic of DeepSeek's inference?