Comment by jrflo
10 hours ago
We really need to stop using $/M tokens as the pricing benchmark. I've found that the number of tokens used tends to be a bigger factor than the listed per token price. The cost per task vs. intelligence curve is really what you care about, and in my estimation Chinese models are just not there. They are focused on benchmaxing and getting the highest raw score they can, rather than efficiency.
The artificialanalysis cost per task chart has DeepSeek as the clear winner and Fable as the clear loser. But I would still pick Fable for some tasks, so that also can't be all there is to it.
But I agree that price per token figure is not great. It seems even the tokens per character can vary between models, so it's basically useless.
Yes, this is already accounted for in many benchmarks, but without deep context of the problem type, the top line pricing is the best starting point.
In my own experience, Fable is more token efficient than opus 4.8 with a higher likelihood of completing tasks correctly or at least with minimal corrective work. Opus regularly struggled to gather the correct context and reason effectively about what it had gathered.
GPT-5.6-sol crushes fable in speed and token efficiency and is clearly superior across many tasks that matter for me.
I also find all models from anthropic after opus 4.6 to suffer from the same ai slop language that long plagued OpenAI and seems to have been reduced drastically in 5.6
Don't forget that you are not really seeing the thinking tokens used - so non-trivial to count them.