Comment by etaioinshrdlu
6 months ago
I've operated a top ~20 LLM service for over 2 years, very comfortably profitably with ads. As for the pure costs you can measure the cost of getting an LLM answer from say, OpenAI, and the equivalent search query from Bing/Google/Exa will cost over 10x more...
So you don't have any real info on the costs. The question is what OpenAI's profit margin is here, not yours. The theory is that these costs are subsidized by a flow of money from VCs and big tech as they race.
How cheap is inference, really? What about 'thinking' inference? What are the prices going to be once growth starts to slow and investors start demanding returns on their billions?
Every indication we have is that pay-per-token APIs are not subsidized or even break-even, but have very high margins. The market dynamics are such that subsidizing those APIs wouldn't make much sense.
The unprofitability of the frontier labs is mostly due to them not monetizing the majority of their consumer traffic at all.
It would be profitable even if we self-hosted the LLMs, which we've done. The only thing subsidized is the training costs. So maybe people will one day stop training AI models.
Profitably covering R&D or profitably using the subsidized models?
He was doing neither. He was using a 3rd party API and has no idea what it costs them to actually run it.
So you're not running an LLM, you're running a service built on top of a subsidized API.