Comment by BoorishBears

Because of how LLMs generalize I'm personally of the opinion we shouldn't have public sets anymore.

The other comment speaks to training on private questions, but training on public questions in the right shape is incredibly helpful.

Once upon a time models couldn't produce scorable answers without finetuning on the correct shape of the questions, but those days are over.

We should have completely private benchmarks that use common sense answer formats that any near-SOTA model can produce.