Comment by phillipcarter
2 days ago
My trust factor is gone with Meta right now. Has there been any independent analysis to confirm they didn't cheat on benchmarks again?
2 days ago
My trust factor is gone with Meta right now. Has there been any independent analysis to confirm they didn't cheat on benchmarks again?
Can't we use it even if we don't trust it?
I don't see a reason to use something that is, at best, about as good as models that are produced by labs who don't have a reputation of cheating.
They cheated again: https://news.ycombinator.com/item?id=48847019
No they didn't. Please read: https://www.anthropic.com/engineering/infrastructure-noise
Thanks for the read. It seems to confirm that resource limits are an important factor for terminal benchmarks:
> The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.
> An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret.
So changing the resource limits changes the benchmark. Yet their score table claims their score to be for Terminal-Bench 2.1, not Terminal-Bench 2.1 with raised limits.