Comment by retinaros
7 hours ago
that is for sure what everyone does. also they train on evals with the datasets that they would be bench against.
7 hours ago
that is for sure what everyone does. also they train on evals with the datasets that they would be bench against.
What do you mean by this? We don’t train on evals, and if we did I’d quit on the spot.
(The loose version of this that’s true is that there may exist eval data contamination in pretraining. This is a hard problem to fully solve for multiple reasons.)