Comment by GodelNumbering

There are also a lot of fake results out there on Terminal Bench 2 for different reasons (although the great team behind it Ryan/Alex et al, recently cleaned up a lot of dodgy submissions). A lot of labs publish the results by modifying timeouts or hardware config which effectively bypasses what is being tested in certain tasks. Then there is harness level cheating, models reward hacking and more...

In fact, one thing that still bothers me after months is the gpt-5.5 official submission. This task in particular https://www.tbench.ai/leaderboard/terminal-bench/2.0/codex/0...

The task has the following timeouts (https://github.com/harbor-framework/terminal-bench-2/blob/ma...).

[verifier]

timeout_sec = 1200.0

[agent]

timeout_sec = 1200.0

[environment]

build_timeout_sec = 600.0

Which means no agent should take more than 3000 seconds doing it. Two out of five attempts in the link above took well over 3000 seconds (75min and 80 min respectively). Even though they failed, the fact that they ran that long is sus.

Goodhart’s Law at work