Comment by yorwba

Their problem is really that individual benchmarks are not that informative. Bad enough models all perform at chance level, good enough models all saturate the benchmark. And the number of tasks is often low enough that differences of several percentage points might just be noise. Artificial Analysis' approach of aggregating scores in a weighted sum is largely just choosing which noise to amplify more.

I prefer Epoch's https://epoch.ai/eci methodology as they explicitly model benchmark saturation and also provide error bars for their estimates.