Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.
Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.
[0] https://deepswe.datacurve.ai/
[1] https://cognition.com/blog/frontier-code-1.1
I've generally found DeepSWE[0] to be pretty true to reality.
[0]: https://deepswe.datacurve.ai/
https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)
1.1 seems a lot better than the original release, which was a bit hyperbolic. excited to see the team keep iterating.
strawberry
Why is this a problem? Its like asking a person how many elder futhark runes are in the word strawberry.
Unless you want to tack on bpe enconding table to every llm context its pointless
FrontierBench
do they have a website? I have found only paper PDF and it seems more general than SWE