Comment by aarondong

Not just compute for OAI, GPT-5.6 is more token efficient across the board vs the Anthropic equivalents: https://artificialanalysis.ai/models?intelligence-index-toke...

No wonder why Tibo can afford to hit the reset button liberally.

I also suspect there is a price fixing agreement between all of the inference providers for Claude (such as Amazon, Anthropic, Microsoft, etc).

There is a blog post waiting to be written (that I won't write) about the size/effort tradeoffs, and particularly how small models get some surprisingly good results with lots of turns and reasoning.

DeepSWE will let you chart turns taken or tokens used, and FrontierCode will chart tokens. If you use that, you can see Sol high and Terra max get about the same DeepSWE number, but Terra max takes twice the turns. Luna max scores a smidgen lower with even more turns.

Smaller models relying on lots reasoning may "scale down" better on easier tasks, because unlike size, reasoning effort is dynamic: the model can see the task looks easy and stop. On DeepSWE, the cost curves for the three 5.6 models are almost on top of each other, but on FrontierCode Extended, the version of FrontierCode with the most everyday tasks in the mix, there's a spread of costs at the ~55% level.

The recent Laguna S 2.1 model (118B, 8B active) puts up surprising coding numbers for its size, and the lab behind it specifically credits its "way of working (persistence, verification, willingness to backtrack)". Some other open models that folks report getting good mileage out of seem to get there partly by throwing a lot of reasoning at the problem.

There is a little bit of a question, if some models rely on getting it wrong a bit more at first and external checks catching the problems, of whether they're also more frequently getting things wrong they can't self-verify (say, quality of UI or API design) and then it falls to the human to find it. Still, getting the results they're getting at all is neat.

Some benchmarks historically favored reporting only on the max variants, maybe because they want to show the frontier? but that is not always what you need for practical decision. (AA has the full effort sweep for Opus 5 and Sol/Luna/Terra at least.) And at least FrontierCode finds Opus 5 taking a hit in performance above 'medium'.

I am not trying to pick a winner here. I'm probably not going to use tiny models on max for everything, but I think it's cool that you can get so much more out of a small model by amping up reasoning, tool use, and persistence.

Luna is the most impressive model released so far by any provider. It's perfect for doing all the low-level tool calling and developing hypotheses.

Terra is great for the humans to talk to.

Sol is really only useful if you need to do more delicate things like synthesis of multiple competing pieces of information.

A system that uses all three variants will massively outperform a system that just uses the biggest model for everything.

Yeah, sol is impressive but IMO Luna is the real standout (and terra is the laggard of the group) for performance/cost

yes; fyi usage limits on the $200 claude sub correspond to at least $1.2k/week in api tokens

Are those actually deployed at scale yet?