Comment by vessenes

I don't understand this conspiracy/suspicion vibe around AI benchmarks -- the best benchmarks get maxed quickly in today's world, especially if the datasets are open. We're in a situation where the AA's of the world show 0-1 point difference between models, so by definition they are NOT good discriminators. These tests NEED improvement.

Do they work with frontier labs? Probably. They should work with them at least a little so that we can get an independent read on the model quality. Do they inject new tests that specifically favor one particular lab? Clearly no, as different labs have moved up over time. Do they inject new tests that help to provide some separation on scoring? God, I hope so.

Nota bene, another change they announced was dramatically increasing the amount of held back test data; this would be a simple explanation for seeing open weights models, all of which are known to distill from frontier labs and benchmarks, drop in the rankings.