Comment by Systemerror7A69
11 days ago
Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.
Who has incentive to improve the quality of the benchmarks?
AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.
> they would love the perfect benchmar
That right there is the problem.
4 replies →
The people who have no power over the situation: users.
Users always have the choice to ignore benchmarks or run their own
4 replies →
The benchmark makers and anyone who relies on benchmarks
depends, are they optimizing for accuracy or something else?
Very much disagree, to me it clearly shows the gap between Open Source and the frontier. You can not do real code work with open source models right now.
Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.
> Very much disagree, to me it clearly shows the gap between Open Source and the frontier.
Yes, this is true.
> You can not do real code work with open source models right now.
This is false, or you're using "real" as a loophole. You can absolutely do real coding work with open source models, right now, right this second.
I'm doing so today using Qwen 3.8 Flash Next running locally, and GLM 5.3 Flash hosted.
We could do real work with Opus 4.5, 4.6. These models (for coding) are more than good enough to do work that Opus 4.6 could do, and in some ways are superior.
Even if frontier models are much stronger than open weights, strong disagree on this. You very much can get real work done using open weights models.
> You can not do real code work with open source models right now.
I only use open weights and get plenty of real work done. Turns out most real tasks are not that hard, rather they are tedious.
People who use mythos for everything also say you cant get real work done with opus.
You just adapted to niceties of your chosen model.
[dead]
This DEFINITELY squares with my anecdata that Gemini 3.8 Flash is not even remotely close to Opus 5.5 on the types of problems DeepSWE CLAIMS to be benchmarking.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
Something seems amiss.
We recently benchmarked our AI code review agent (just pi with some custom prompts based on info from our context layer). Previously we used Gemini flash 3.8 as it was capable and affordable. However, haiku 5.5 just blows it out of the water. It's more than 10x cheaper, 2x faster and finds 3x as many real issues.