Comment by simonw
19 hours ago
My point with the $2,180.16 thing is that the price for consumers like myself is heavily discounted... but the price for enterprise companies is not discounted.
My usage is therefore a useful indicator of quite how much those enterprise companies may be spending on tokens, given the new pricing scheme.
If enterprise companies were still getting the same discounts that I get myself I would not have written this article.
(I had to dig into your margin figure - looks like you calculated 5.1% as 559000000 / 10900000000 * 100 but that $559M "adjusted operating profit" figure includes training costs, where usually when we talk about margin on inference we're not including those since those costs are fixed, margin calculations make more sense against the variable costs of serving a token.)
When you have to train a new model every few months to stay competitive, discounting that cost is rather dubious.
They key difference here is that training costs are fixed. If you train a model for $100m dollars, how much of that training fee should you allocate to each token that the model serves?
It's impossible to know, because you don't know how many tokens total will be served by that model until you retire it at some point in the future.
So you can't say "1,000,000 tokens costs $X in inference and $Y in training" because $Y is not possible to correctly calculate.
So, if you want to have a productive conversation about "margin on inference", it's sensible to look at the cost of serving the tokens independently of the cost of training the underlying model.