Comment by atleastoptimal
19 hours ago
Counterarguments to that premise:
1. Intelligence per dollar spent/token output seems to be increasing rapidly. Opus 4 was $15/75 per million input/output tokens when released. Opus 5.5, which is significantly better, is $4/15. Sure it cost more money upfront to train Opus 5.5, or the bigger model (Fable 5.5) which tuned it, but once it is reached, there are massive diffusive effects.
2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
3. The search space is too large to assume that improvement in AI assisted research is just models dumbly trying every combination until something works. For every additional bit searched, the space doubles, so even a trivially small combinatorial exploration incurs prohibitively large costs. Intelligent searching which continuously prunes that space along dimensions of effectiveness is not a trivial skill and is among the emergent accoutrements of these smarter models.
> 2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
But they are essentially fighting for smaller and smaller part of the pie. The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is.
Sure there is inertia and the usual corporate enterprise nonsense of "we only buy stuff from other big companies regardless the cost" that keeps the licenses going but we're quickly getting into point where you don't need any of the frontier models to do majority of the tasks
I don't fully get this line of argument. I am a massive advocate for open source... everything basically, but especially in ML. And open models are making incredible strides. But I'm not sure that models getting good enough for most tasks, making the absolute frontier models (Astra/Fable class) unnecessary for most people, puts the frontier labs in a fundamentally disadvantaged position.
I mainly think this because frontier labs already try to compete at this tier. For example, 6.1 Sol has identical pricing to Kimi K3.1, but with a lower price per task due to Kimi's verbosity. It seems to score roughly the same on most benchmarks, with some (like Artificial Analysis) putting 6.1 Sol significantly higher. If you need to step down, 6.1 Terra is also cheaper than K3.1 or K3 and performs well relative to their benchmarks. Opus 5.5 and much of Anthropic's recent push seems to confirm this as a major area of focus as well.
I still use a lot of open models and harnesses, but I don't know if cost competitiveness (at least not with the current crop of open models) will ultimately drive a huge chunk of average users away.
Things could change, but fundamentally, large labs aren't precluded from training more cost effective models if there is significant demand for them, which is what we're seeing now.
You did not really address the argument.
6.1 Sol is not a “frontier model” and does not command the pricing of one. The reason it exists is because of competition from non-frontier models that are “good enough”.
In fact, you kind of made the same argument. As “good enough” models commoditized, the use cases requiring frontier models and frontier model pricing will shrink.
And, even if performance is not as good, many will prefer open models.
2 replies →
Let's just ignore the many practical advantages of open and self-hosted models...
1 reply →
> The better open models are the less you need frontier models
True in theory, but in practice hardware availability and cost will keep open source impractical for years to come.
It will be impractical to self host, but not to get one that's served from other providers (which won't have to recuperate loses from training).
If your moat is "Our vendors aren't expanding fast enough to supply our competitors!", you don't have a moat.
It's already very practical lmao
2 replies →
Don't you mean the opposite, that intelligence per dollar is increasing?
yeah I reversed it when revising my phrasing
> Intelligence per dollar spent/token output seems to be increasing rapidly.
I would argue this is more due to market economics than actual inherent cost. Opus 4 was when the open models started catching up. The R1 bootstrapping chain of thought paper via RL paper was released a couple of months before Opus came out and the open models started to significantly catch up after that. The duopoly was effectively broken and they lost their price setting power.
What is the unit of measure for “intelligence per dollar”, and how do you measure it? Without a concrete definition, that first assertion is not even wrong.
I would also like to point out that “profitable on inference” is similarly meaningless, because it’s based on a figure that arbitrarily discards whatever bits of their operating costs they need to to get to the claim that they’re profitable on inference. Potentially including ones that can’t be ignored because they’re every bit as much a part of COGS as the actual inference is.
>What is the unit of measure for “intelligence per dollar”, and how do you measure it?
It's going to depend on the task (and this is prone to cheating), but you can get a decent idea by looking at task benchmarks and the cost of completing them.
I was inclined to agree but Anthropic's latest S-1 statement only includes a paper loss due to their value going up. They're objectively making money now, the only question is can they keep it up and can they grow enough to be able to pay the bills they've signed up for in future?
But benchmark performance is not the same thing as intelligence.
Which is a really important distinction. One measures specific points on the jagged boundary. The other is a word that blithely paints it all with a single brush, and sweeps the jagged boundary problem under the rug in the process.
It really is important to keep track of that. Whenever we talk about “intelligence” as if it were a well-defined (let alone quantifiable) thing, we fall into the kind of wooly thinking that lets anyone say pretty much anything. We abandon the realm of grounded statements in favor of pure vibes.
1 reply →