Comment by minimaxir
13 hours ago
DeepSeek v4 Flash recently had a similar "more reasoning is cheaper" curve. It's a fun counterintuition.
13 hours ago
DeepSeek v4 Flash recently had a similar "more reasoning is cheaper" curve. It's a fun counterintuition.
It’s not that different than a lot of real world economies. Often paying for someone or something with better quality can reduce total costs. You have less failures, less mistakes, so on, so while the expertise or quality of the product is higher than cheaper solutions, they can be more reliable and over time ultimately cheaper.
The question I have is how far back that curve can go without relying on economies of scale to just drag all the points back to the left. And without overfitting a specific metric that I don’t need (like this test).
What is the intuition. Higher quality turns due to more reasoning results in significantly fewer turns taken?
Yep. In particular, ARC-AGI-3 is a series of games where if you fail, you keep trying again (until eventually hitting a timeout). So the sooner you succeed, the sooner you stop spending tokens retrying. If it was a benchmark where everyone got one attempt with no retries, you wouldn't see it bend backward.
Yes, in theory.