Comment by ramon156
1 year ago
Wouldn't be surprised if similar methods are used to improve benchmark scores for LLM's. Just make the LLM respond correctly on popular questions
1 year ago
Wouldn't be surprised if similar methods are used to improve benchmark scores for LLM's. Just make the LLM respond correctly on popular questions
Oh for sure. The questions for most benchmarks can be downloaded on hugging face.
I thought this is why most of the benchmarks have two parts, with one set of tests public and the other set private?
In an ideal world yes, but in order for LLM authors to provide the evals they need to access the private set (and promise not to train on them or use that to influence eval/training methods).
Either the eval maintainers need to be given the closed source models (which will likely never happen) or the model authors need to be given the private evals to run themselves.
14 replies →
Because of how LLMs generalize I'm personally of the opinion we shouldn't have public sets anymore.
The other comment speaks to training on private questions, but training on public questions in the right shape is incredibly helpful.
Once upon a time models couldn't produce scorable answers without finetuning on the correct shape of the questions, but those days are over.
We should have completely private benchmarks that use common sense answer formats that any near-SOTA model can produce.
Plus rankings like lmsys use a known fixed system prompt
Looking forward to LLMgate