Comment by enugu
1 year ago
> Either the eval maintainers need to be given the closed source models (which will likely never happen)
Given that the models are released to public, the test maintainers can just run the private tests after release, either via the prompts or via an api. Cheating won't be easy.
Closed source in this context literally means that they are not released.
They dont need them to be released(in the sense that you have a copy of the binary) to evaluate the model. The costly model training is useless unless access is given to people who pay for it.
The models of Open AI, Claude and other major companies - are all available either for free or a small amount(200$ for OpenAI Pro). Anyone who can pay this, can run private tests and compare scores. So, the public does not need to rely on benchmark claims of OpenAI based on its pre-release arrangements with test companies.
> Anyone who can pay this, can run private tests and compare scores.
Yes, by uploading the tests to a server controlled by OpenAI/Anthropic/etc
1 reply →