Comment by vb-8448
11 hours ago
> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing".
No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models.
Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)
> No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models.
Still useful; "are the labs marginally profitable just on the marginal inference costs?" is still a useful question to answer. After all, if they aren't even profitable on inference in isolation, then we can expect to see large price increases.
If they are able to turn a marginal profit on inference alone, then perhaps the price increases won't be so severe (or perhaps they expand the time between generations so that they spend less on training but take longer to complete training).
"Are the labs profitable at all?" is, of course, a much more useful question, but that doesn't mean that the first question is completely useless.
I wasn't saying it isn't useful, I was saying that you cannot infer even the marginal cost of closed models.
The K3 maths can turn true only if the models size is roughly the same and the labs didn't find any better way to run inference at scale.
We know labs make money on inference, and we know they lose a lot of money on inference+training.
> We know labs make money on inference
We don't really know that, for OpenAI and Anthropic. We suspect that, but as far as I know, even they have stopped claiming that they are profitable on inference.
1 reply →
Just out of curiosity, based on what we know for sure they(OAI+A) make money on pure inference and lose on inference+training?
1 reply →
> Without training cost you can infer only the marginal cost of serving this kind of models.
Which is by far the most interesting number of the two.
> Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)
If you get close in output quality, then does that matter?
> If you get close in output quality, then does that matter?
When you're trying to estimate/infer the costs of serving the tokens and even include the cost of training the weights in order to output tokens then yeah, why wouldn't that matter?
That only matters if you are an investor not if you are a consumer.
2 replies →
> Which is by far the most interesting number of the two.
Only if you don't have to continuously train new models, and you are not at a runway risk.
Training cost is directly impacted by inference cost nowadays. Most of the gains come from RL these days, and that is highly dependant on inference (~7:1 inference:training in units of compute). That's mainly because you want many roll-outs for each training scenario.
Of course inference efficiency is dictated by model architecture, size, etc. You can still guesstimate some of those and have an idea about cost/serve at several size tiers.
That isn't really relevant to GP's point. We still don't know what training costs because we don't know how much RL is done.
Gross margins are insanely important, possibly the most important single metric if for some reason you were forced to choose one.
Only if you assume that at some point, for any reason, there will be "the model" that doesn't need costly retraining.
I guess this is one of the reason Anthropic i so "active" for asking for a development break.
"I want my competitors to stop competing" is an interesting ask for someone who's currently charging the highest prices in the entire industry.
>No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models.
Are you talking about Kimi's training cost or the training cost of the model(s) that Kimi distilled?
Because Moonshot didn't even incur the majority of the training costs either
I believe they are talking about the closed models' training costs.
I other words, the providers that will be offering K3 inference don't have any training costs to offset, so they are only charging for the inference itself. OAI/Anthropic would need to offset their R&D and training costs in order to not be selling API access at a loss.
> the training cost of the model(s) that Kimi distilled?
The distillation process involves getting conversation traces from the model you are distilling from, and then training your model against them.
You still have to do the training!