Comment by sshh12
1 year ago
In an ideal world yes, but in order for LLM authors to provide the evals they need to access the private set (and promise not to train on them or use that to influence eval/training methods).
Either the eval maintainers need to be given the closed source models (which will likely never happen) or the model authors need to be given the private evals to run themselves.
So the entire benchmark scheme is worthless?
Well it depends on how you define worthless. For you as an individual to ascertain truth, it may be useless. To build up a bloated AI Enterprise stock value. For false consensus narrative scripting. Very valuable.
Benchmarks have been close to worthless since Nvidia cheated 3DMark benchmarks by adding benchmark detecting code to their drivers in the early 2000s [1].
[1] https://www.eurogamer.net/news280503nvidia
That's a bit dramatic. One person cheating doesn't mean everybody is cheating. Cycling didn't become pointless after Lance Armstrong was caught doping. But it may be the tip of the iceberg and warrant a change in methodology or further investigation.
3 replies →
It’s essentially worthless for you, as a consumer of them. The best way to see which one works best is to give a bunch of them a try for your specific use case
Not if you have a private eval you use for your own use cases.
> Either the eval maintainers need to be given the closed source models (which will likely never happen)
Given that the models are released to public, the test maintainers can just run the private tests after release, either via the prompts or via an api. Cheating won't be easy.
Closed source in this context literally means that they are not released.
They dont need them to be released(in the sense that you have a copy of the binary) to evaluate the model. The costly model training is useless unless access is given to people who pay for it.
The models of Open AI, Claude and other major companies - are all available either for free or a small amount(200$ for OpenAI Pro). Anyone who can pay this, can run private tests and compare scores. So, the public does not need to rely on benchmark claims of OpenAI based on its pre-release arrangements with test companies.
2 replies →