Comment by rao-v

In a real job, you would be allowed to see the test case that failed and tweak your code (or more likely the poorly written test).

If you let a modern LLM do even the first, they’d crush this specific benchmark.

What is interesting is understanding how LLMs are able to beat 70+% on this benchmark or getting some of the poorly framed questions right? Are they implicitly learning the test writers style? Are the solutions leaking into their training set?

Perhaps reassuring is that even Fable stalls out at ~72% (on the hidden set which OpenAI did not run this analysis on), so perhaps training on the bench is not happening in anything but the most indirect ways.

I care a lot because small open models can never learn idiosyncrasies like this, so I really want good ways to judge models fairly.

EDIT: Humm OpenAI is muddying the water a bit. Only 20%ish of problems are broken in ways that are unfair to the agent, 4-10% are broken in favorable ways, so the benchmark ceiling is probably closer to 80-85%