Comment by saberience 2 days ago SWE Bench Pro is also gamed and shouldn't be trusted. 2 comments saberience Reply mnicky 2 days ago Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is. Sabinus 2 days ago Why that benchmark in particular?
mnicky 2 days ago Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
Why that benchmark in particular?