Given it was made by cognition (team behind devin flop) who now just got to wait out until claude and gpt5 basically do all of the work for them - not very. When you read about it, the framework is highly subjective. Which very quickly becomes a problem because its based on heuristics that probably change a bunch with a better code model.
i worked on one of the benchmarks typically found in new model releases
this benchmark looks very good from the methodology. a cog researcher checking the data themselves is very high signal (not scaleable so don't take the benchmark as gospel, but directionally good)
It's a relatively new benchmark but from what I can tell it has serious cred behind it. I assume it will be picked up as part of the standard suite of CS-related benchmarks soon enough.
Yeah, right. If this benchmark was truly developed in an independent manner, and the timing just “lined up”, how did Anthropic even know to include results in their model release documentation the day after the benchmark is revealed? It seems like there must have been some collaboration or influence from Anthropic behind the scenes.
Cognition did well in documenting their approach [1].
TL;DR - they worked with OSS project maintainers to build tasks. They score models based on whether a PR is mergeable. All tasks are graded by a human researcher. SoTA models have hill-climbing to do which raises the bar and inspires confidence. I'd say it's legit.
Did you read the blog post? They compare to deepswe and call it out as the worst one for false positives (failed, but the benchmark assessed it as correct). It also has less language variance.
Given it was made by cognition (team behind devin flop) who now just got to wait out until claude and gpt5 basically do all of the work for them - not very. When you read about it, the framework is highly subjective. Which very quickly becomes a problem because its based on heuristics that probably change a bunch with a better code model.
the subjective framework is exactly why its good
prior bms relied mostly on unit tests or synthetic judges which are easily benchmaxxed, which leads to nobody trusting benchmarks
we need people manually checking the data for good code quality
i worked on one of the benchmarks typically found in new model releases
this benchmark looks very good from the methodology. a cog researcher checking the data themselves is very high signal (not scaleable so don't take the benchmark as gospel, but directionally good)
It's a relatively new benchmark but from what I can tell it has serious cred behind it. I assume it will be picked up as part of the standard suite of CS-related benchmarks soon enough.
Seems like it literally popped up yesterday with the express purpose of building hype for this release.
And notable absence of DeepSWE benchmark where they do badly, but somehow a benchmark that was published yesterday is in this announcement.
Exactly.. a bit of a red flag for me..
team member here - we had been working on frontiercode for ~6-7months. timing just lined up
Yeah, right. If this benchmark was truly developed in an independent manner, and the timing just “lined up”, how did Anthropic even know to include results in their model release documentation the day after the benchmark is revealed? It seems like there must have been some collaboration or influence from Anthropic behind the scenes.
1 reply →
i doubt it, cog wants coding agents to be better because it directly improves their product
they aren't married to a particular lab, most of their usage is their in house model i believe
what incentive does Cognition have for doing this? seems like complete nonsense speculation on your part.
With billions/trillions of dollars floating around, is it hard to imagine benchmarks could be biased?
I think it's safe to assume everything AI related is heavily biased until proven otherwise. Just like in pharma.
4 replies →
Cognition did well in documenting their approach [1].
TL;DR - they worked with OSS project maintainers to build tasks. They score models based on whether a PR is mergeable. All tasks are graded by a human researcher. SoTA models have hill-climbing to do which raises the bar and inspires confidence. I'd say it's legit.
[1]: https://x.com/cognition/status/2064061031912288715
It's an unacademic benchmark by a failed VC startup clawing for relevancy.
DeepSWE is the benchmark you want to actually look out for. Only one that aligns with actual user reported results from trying the models.
Did you read the blog post? They compare to deepswe and call it out as the worst one for false positives (failed, but the benchmark assessed it as correct). It also has less language variance.
I mean yes that is what you'd say if you were writing a blog post about your new benchmark.
1 reply →
[dead]