Comment by d2p
2 hours ago
I clicked through and it showed Qwen at the top at 55.4 compared to 55.3 for Opus Max. I have a screenshot.
Then I clicked away and back, and now it goes Qwen second, with 58.4, to Opus Max at top with 59.2.
I have screenshots of both. The description above the chart is the same in boh cases:
> Artificial Analysis Agentic Index > Represents the weighted average of agentic capabilities benchmarks in the Artificial Analysis Intelligence Index (GDPval-AA v2, ³-Banking)
What happened? How can the scores change so much in a few seconds?
They JUST updated their methodology:
https://artificialanalysis.ai/methodology/intelligence-bench...
Edit to provide AA's article explaining it:
https://artificialanalysis.ai/articles/artificial-analysis-i...
Fixed the result, eh? In both senses of the word.
Can someone please explain what changed, when it happened, and whether it was surreptitious?
I have been suspicious of these AI leaderboard sites for some time now, and this only increases that suspicion.
In that case they should clearly label that this is a new benchmark.
What was the change?
They should probably freeze the results before publishing.
Welp. That didn't last long
Same, they just updated it. Hacker news effect?