Comment by msdz

It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process).

“Good” benchmarks to gauge development skills at the moment seem to be:

DeepSWE [0] by Datacurve

FrontierCode [1] by Cognition

And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.

But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.

[0] https://deepswe.datacurve.ai

[1] https://cognition.com/blog/frontier-code

[2] https://xcancel.com/i/article/2064210146558136827