It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process).
“Good” benchmarks to gauge development skills at the moment seem to be:
DeepSWE [0] by Datacurve
FrontierCode [1] by Cognition
And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.
But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.
First look what models are worse in a set of self selected benchmarks.
Second, compare to older versions of competitor s models.
Still does not look good? Compare to own previous models.
Not much moat, incremental improvements, cherry picking models to compare.
To be fair, seems more correct to compare against similar strength models if your main edge is pricing.
At this point comparing to Gemini is a free Bingo space.
Wait to the exact moment your model is ahead on at least N benchmarks then publish.
Anyone deep in the AI realm know which is the gold standard benchmark for coding?
It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process).
“Good” benchmarks to gauge development skills at the moment seem to be:
DeepSWE [0] by Datacurve
FrontierCode [1] by Cognition
And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.
But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.
[0] https://deepswe.datacurve.ai
[1] https://cognition.com/blog/frontier-code
[2] https://xcancel.com/i/article/2064210146558136827
They're being called "trust me bro benchmarks" for a reason ( ･ั ﹏ ･ั )