Comment by GodelNumbering
4 hours ago
As of their latest update (AA's), I find their index hard to trust, and a bit shady. 60% of the benchmarks contributing to the AA index are now private, nobody can replicate them. And they explicitly say that they offer benchmark consulting to labs. This is a serious conflict of interest.
The latest update knocked down only openweights models from the leaderboard.
I agree that they are a bit shady, but private benchmarks are the only way to prevent benchmaxxing
Can’t blame them. Silicon Valley is expensive.