Comment by simonjgreen
12 hours ago
This might be the first time I’ve favourited a post without upvoting it.
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
From my rough recollection, all 3 of the past updates all had the effects of decreasing the score of open weight models.
One was rushed and same-day updated as GLM 5.3 (iirc) got added; of course its main change to rankings was lower GLM scores.
Such releases often show weaknesses if the intelligence index, because the models that are ranked jointly do not have the same percieved level of quality.
So instead of the narrative of benchmaxing, in truth it's benchminimizing of competitors by the U.S. oligarchy?
Their problem is really that individual benchmarks are not that informative. Bad enough models all perform at chance level, good enough models all saturate the benchmark. And the number of tasks is often low enough that differences of several percentage points might just be noise. Artificial Analysis' approach of aggregating scores in a weighted sum is largely just choosing which noise to amplify more.
I prefer Epoch's https://epoch.ai/eci methodology as they explicitly model benchmark saturation and also provide error bars for their estimates.
I noticed last night that openai had fallen way down the index, and I allowed myself to think for a minute that maybe AA aren't as biased as I thought. Then a few hours later they push this fix which sees openai rise again.
This update is even worse than usual because it now includes two private sources, data which we can't reproduce if we wanted to do.
Also note that Anthropic is the only AI company actually caught red handed cheating at evals. Teaching their models to memorize test answers and lie about it, using their hidden reasoning tokens to hide the deception [0]. Yet we see no comment from AA about this. No desire to figure out which tests are tainted. Hopefully Trump bans this fake intelligence index.
[0] https://arxiv.org/abs/2608.09867
Is that last bit supposed to tongue in cheek? Sarcasm is difficult to read through text.
Trump bans artificial-intelligence term.
https://thehill.com/homenews/administration/6175633-trump-ba...
Devil's advocate: Their scoring system is itself a meta-benchmark.
What happens when you release a new benchmark? Models magically start scoring better against it.
How do you overcome this? Release a new benchmark and start the process over again!
All other things being equal, they should retarget their prices if models start gaming the process to top their index.
Has this happened? Maybe. Is their new scoring system biased towards the American frontier models? Quite possibly.
I have my suspicions, but they have led me astray before. I will continue to be cautiously cynical, though.
small (700b) open-weights models are great at coding and tool calling but lack world knowledge. because the parameter difference is so huge T to B, of course they also are not as intelligent.
this is i think the difference in this new analysis.
doesnt mean deepseek is bad or useless.
Fun fact: By changing the weights of the input you can change the output as you want.