Comment by cortesoft
1 year ago
I thought this is why most of the benchmarks have two parts, with one set of tests public and the other set private?
1 year ago
I thought this is why most of the benchmarks have two parts, with one set of tests public and the other set private?
In an ideal world yes, but in order for LLM authors to provide the evals they need to access the private set (and promise not to train on them or use that to influence eval/training methods).
Either the eval maintainers need to be given the closed source models (which will likely never happen) or the model authors need to be given the private evals to run themselves.
So the entire benchmark scheme is worthless?
Well it depends on how you define worthless. For you as an individual to ascertain truth, it may be useless. To build up a bloated AI Enterprise stock value. For false consensus narrative scripting. Very valuable.
Benchmarks have been close to worthless since Nvidia cheated 3DMark benchmarks by adding benchmark detecting code to their drivers in the early 2000s [1].
[1] https://www.eurogamer.net/news280503nvidia
4 replies →
It’s essentially worthless for you, as a consumer of them. The best way to see which one works best is to give a bunch of them a try for your specific use case
Not if you have a private eval you use for your own use cases.
> Either the eval maintainers need to be given the closed source models (which will likely never happen)
Given that the models are released to public, the test maintainers can just run the private tests after release, either via the prompts or via an api. Cheating won't be easy.
Closed source in this context literally means that they are not released.
3 replies →
Because of how LLMs generalize I'm personally of the opinion we shouldn't have public sets anymore.
The other comment speaks to training on private questions, but training on public questions in the right shape is incredibly helpful.
Once upon a time models couldn't produce scorable answers without finetuning on the correct shape of the questions, but those days are over.
We should have completely private benchmarks that use common sense answer formats that any near-SOTA model can produce.