← Back to context

Comment by pvillano

10 days ago

I just want a benchmark that gives zeros for cheating

'Cheating' is more of a spectrum than a binary choice.

Some of my favorite personal benchmarks are brainteasers with subtle changes that test whether the model 'cheats' by returning a stock answer or actually reads the whole prompt and reasons through the premises given. E.g., the usual liar/truth-teller riddle but modified with 'One guard always tells the truth, the other never lies.'

All models parrot some variation of 'Ah, yes, the classic liar/truth-teller riddle,' in their thinking traces, but only the better ones will hedge their final answer with some variation on 'The usual answer is X but if you really meant that they both tell the truth, the answer is Y.'

That doesn't mean they cheated by acknowledging that something very much like the prompt was in-distribution for them. As long as they get the job done without relying solely on training data, it's all good.

  • Wikipedia defines cheating as actions designed to subvert or disobey rules in order to obtain unfair advantages without being noticed. This is the definition I use.

    What you've described is not applicable to this definition of cheating.

    I care about agents reading the solution from the git history, breaking test harnesses, or using disallowed external tools, because these prevent SWE bench from being an accurate predictor of real world usefulness in fixing bugs