Comment by seizethecheese
2 hours ago
Opus is still first in Intelligence Index followed by Fable, GPT 5.6, Kimi K3 then Qwen 3.8 max. https://artificialanalysis.ai/#intelligence
Our leaderboard combines Arena ELO, AA Intelligence index, latency and speed and goes: #1 Opus 5 #2 Kimi K3 #3 Qwen3.8 Max #4 GPT 5.6 Sol
Source: http://pellmell.ai/leaderboard.
This jumps around a lot based on the top throughput and latency of whatever provider happens to be best at the moment.
All these "intelligence" benchmarks miss something extremely important when using an LLM in a code-agent harness: How it communicates with you about what it did.
Opus-5 is practically unusable (for complex tasks) in this sense - its updates are voluminous, and dense with cryptic language (there are numerous reddit threads complaining about this, so it's not just me). I often have to ask it to re-state concisely in plain terms.
For a fairly gnarly task, after fighting with with Claude-Code + Opus-5, I ported my session to Codex + GPT-5.6-sol, and it was like a breath of fresh air.
Arguably a key aspect of intelligence is concise, clear communication, and current benchmarks miss that, at least as far as I'm aware. I would think some arena-type benchmarks where humans rate responses would measure this, though I'm not sure which those are.
Is that what it feels like when the models get smarter than us?
No the models are just ass at communication without being directed.
Try asking them to make useful diagrams for some stuff in a codebase, out of the box without excessive hand holding they don't make good choices about what's worth communicating and how to do it.
You see this in their pointless frontend copy all the time too.
"There is a view in some philosophical circles that anything that can be understood by people who have not studied philosophy is not profound enough to be worth saying. To the contrary, I suspect that whatever cannot be said clearly is probably not being thought clearly either."
A smarter model would know how to communicate with you correctly, and not just throw jargon it has just invented at you without explaining it.
5 replies →
No, a smart model should also give a concise executive summary, "brevity is the soul of wit".
2 replies →
Guess that's the exact point of the "intelligence" benchmarks
GPT 5.6 has similar language quirks that makes its comments nearly unusable.
I wonder if this is a side effect of MoE models — they can write excellent prose, but not simultaneously with writing code.
Eh I don't know, I care whether it gets the job done and I can see the difference when I review the code, not how well it needs to explain the code to me, I can just read it myself.
Damn I thought it was my extra instructions, I swear everything it writes is in some shorthand with direct references to variables that literally nobody could figure out unless you literally just wrote that code 5 minutes ago. I had it stop writing comments altogether cause it was always four lines of complete and utter nonsense, and it doesn't even obey that rule half the time. Despite doing an extensive back and forth to make a complete plan, 5 seconds into the implementation it changes its mind and makes another assumption, adding some extra thing that tends to break the entire approach and needs follow-ups to repair or cleanup. Instruction following is basically non-existent compared to Fable, it just does whatever the fuck it wants.
Everything is load-bearing with 3 measured blockers.
1 reply →
It takes 2 minutes to fix Opus 5
https://code.claude.com/docs/en/output-styles
lol. I am being downvoted for trying to help people out.
This community is pure trash.
It seems like if latency is having such a big effect that it's changing the winners, maybe your tests are awful and shouldn't be so latency dependent?
I mean, I get it: how fast a model responds is relevant. But a test that changes second by second is far less relevant than a test that tells you how smart the model is, and accounts for latency in some way that isn't constantly changing the result.
Latency isn't changing the results for the coding index or arena ELO, but neither of those take latency or throughput into account, so we added those to our leaderboard as score components.
Latency and throughput matter a ton as a user, so I think it's actually totally defensible for a leaderboard to bounce around a lot as these numbers change. The best model to use changes a lot based on these!