Comment by ssivark
3 hours ago
Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).
But the end user doesn't care about flops for closed models. All they care about is how much it ends up costing them.