Comment by rbehrends

Neither is the primary cause, I suspect. If you look at the coding index alone, you see more plausible results from a software engineering perspective (in particular, Terra no longer outperforming Fable) [1].

That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2].

The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.)

Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3].

I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are.

[1] https://artificialanalysis.ai/#coding-agents-tabs

[2] https://artificialanalysis.ai/evaluations/terminal-bench-sci...

[3] https://artificialanalysis.ai/#cyber-index-tabs