Comment by saberience

2 days ago

The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.

4 comments

saberience

Reply
mnicky  2 days ago

SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.

  • saberience  2 days ago

    SWE Bench Pro is also gamed and shouldn't be trusted.

    • mnicky  2 days ago

      Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.