Comment by famouswaffles
3 hours ago
Like the commenter above specified, the best way to satisfy the grader is to get the answer key, regardless of how clever you are, especially when you realize lots of these benchmarks have flaws (i.e wrong answers, overly restrictive grading etc).
Was the answer key stored on huggingface's servers and this is public knowledge?
In OpenAI's writeup (https://openai.com/index/hugging-face-model-evaluation-secur...) they explain that the model initially spent its efforts obtaining internet access in an attempt to cheat on the evaluation. Once it got that internet access, it was able to do research that lead it to believe that HuggingFace had infrastructure that hosted the evaluation and potentially had the answers.