Comment by jrflo
19 hours ago
We really need to stop using $/M tokens as the pricing benchmark. I've found that the number of tokens used tends to be a bigger factor than the listed per token price. The cost per task vs. intelligence curve is really what you care about, and in my estimation Chinese models are just not there. They are focused on benchmaxing and getting the highest raw score they can, rather than efficiency.
The artificialanalysis cost per task chart has DeepSeek as the clear winner and Fable as the clear loser. But I would still pick Fable for some tasks, so that also can't be all there is to it.
But I agree that price per token figure is not great. It seems even the tokens per character can vary between models, so it's basically useless.
Wow, you weren't kidding. I looked at their chart, and the cost-per-task for Fable is more than double Sol's. And DeepSeek absolutely stomps. Four cents per-task vs Sol's $1 and Fable's $3.
I might need to check out DeepSeek more. I had no idea the difference was this obscene. Makes me wonder if something's off with the benchmark. A 70x cost reduction vs. Fable seems too good to be true.
Almost as if centuries of passing the imperial exam by either skill or cheating influenced Chinese culture a lot.
While I do agree that cost per task is what customers should care about, and not cost per token. Cost per token is an objective metric. Cost to do a task can vary a lot. Different tasks, different prompts, or just pure randomness nature of models make it a bit harder to define this as an objective metric.
Unless the cost per token is prohobitedly high, people can often try the model out themselves and make a subjective judgement of how effective and efficient is it at solving tasks they usually deal with, using their setup.
Ockbench: https://ockbench.github.io/ show token usage and accuracy, but you have to compare it with other stats (prices and speed).
Yes, this is already accounted for in many benchmarks, but without deep context of the problem type, the top line pricing is the best starting point.
In my own experience, Fable is more token efficient than opus 4.8 with a higher likelihood of completing tasks correctly or at least with minimal corrective work. Opus regularly struggled to gather the correct context and reason effectively about what it had gathered.
GPT-5.6-sol crushes fable in speed and token efficiency and is clearly superior across many tasks that matter for me.
I also find all models from anthropic after opus 4.6 to suffer from the same ai slop language that long plagued OpenAI and seems to have been reduced drastically in 5.6
Don't forget that you are not really seeing the thinking tokens used - so non-trivial to count them.