SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful
5 days ago (github.com)
Quick update here since a lot of folks asked about what slopcodebench is / where it fits in
Slop code bench is unique among benchmarks in that it makes a model work against a progressively disclosed list of features (checkpoints) that it doesn’t know it advance, inheriting the same codebase through the lifecycle of a single multi-checkpoint challenge
I like this because it much more mirrors how real software dev happens, rather than most benchmarks with one off tasks where slop code doesn’t hinder future progress
The original post[1] and the SOL/fable follow up[2] went into a bit more detail on the benchmark and its challenges
1. https://github.com/humanlayer/advanced-context-engineering-f...
Astra writes well (which is why I often like to work with it over Claude), but it tends to produce code golf that's borderline unreadable, like completely omitting line breaks. I thought I had something funky with my config, but it seems to be a known issue.
https://lucumr.pocoo.org/2026/9/7/astra-why/
EDIT: You can search for "tenobrus astra" on X, but link removed.
The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?
I use the frontier models only in plan mode; coding is held out to cheaper models.
I run Paseo as an orchestrator with agent profiles. "Distinguished Member of Technical Staff" -> Fable, plan mode. "End-User Customer Advocate" -> Astra, plan mode. A few others. Basically, these agents (with profile prompts) can only be "consulted" for input on design elements; it's OK if they produce a bit of verbosity here. This has worked fantastically for me, really awesome to see a self-constructed team own the work end-to-end.
A "Director of Engineering" (6.1 Sol or Opus 5.5) also in plan mode organizes the team for the task and then owns coordination and instruction. You can think of this in essence like a "model router".
Coding is handled by Haiku 5.5/GPT 6 Luna/DeepSeek v4.1 Flash. One "Principal Engineer" Sonnet 5.5 or 6.1 Sol for some tasks.
You are deep in it.
You may be larping for no additional benefit vs just using Opus 5.5 and just asking it to implement a feature.
11 replies →
Don’t apologize for x, just don’t post x links.
I think it's the best possible approach. It lets you refer to content that's unfortunately stuck on the site, while still signaling clearly that it's something to avoid if possible and to apologize for if not. Omitting the link just removes information without influencing consensus on whether to use X.
edit: I wish it would let me view the darn screenshot tho :c
3 replies →
X links are fine. People who don't want to visit the site can choose not to visit it.
This current situation where people get shamed for sharing the information until they edit it out and leave obscure "search for___ on X" is infinitely worse.
I removed it, but it's frustrating when the options are someone's (reasonable) post on X, or slop-spam aggregators. I don't really want to drive visits to either, but at least the X poster can get the views. I don't really like xcancel either as it still encourages indirect engagement with X.
Astra is some of the worst code I've seen in terms of slop, so the benchmark is kind of surprising to me. Astra gets things right, but I need Claude clean up its slop all the time. Many times I've cut down Astra changes from 1000+ lines to ~100 lines.
I've noticed the line breaks issue. It's easily fixed with a formatter like prettier and, frankly, that kind of quality-of-life thing is best handled deterministically anyway.
I haven't used prettier much but this is what the prettier docs say about empty lines.
> It turns out that empty lines are very hard to automatically generate. The approach that Prettier takes is to preserve empty lines the way they were in the original source code. There are two additional rules: > > - Prettier collapses multiple blank lines into a single blank line. > - Empty lines at the start and end of blocks (and whole files) are removed. (Files always end with a single newline, though.)
The last time I used astra it wrote a lot of rust code where it would insert almost no empty lines at all. Not before functions, structs, impl blocks, etc., and not inside functions to group statements. Similar to prettier, rustfmt doesn't really insert empty lines.
Edit: hmm maybe I misunderstood your and the parent's comment. Are you talking about missing regular newlines after statements, i.e. multiple statements on one line?
1 reply →
Doesn’t this mess up the context though? I use ruff and my agents get confused if I edit and run ruff out of band. How do people deal with that? I’m always having to remind agents to re-read files because they’ve changed
1 reply →
xxcancel (n.b. dos equis) works:
https://xxcancel.com/tenobrus/status/2096636223414808719
Beware NSFW: redirects to private instances that redirect to porn for monetization
Edit: disable JS to prevent click-jacking
1 reply →
It's Opus specific, but personally I'd never bother with Opus above High, CursorBench shows a clear inflection: https://cursor.com/cursorbench
Is there a reasonable way to run a representative subset myself?
Edit:
> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.
And https://github.com/gabeorlanski/scb-problems/
Yeah and I started with subsets although I did not find them representative, eg I picked 6 challenges for a subset and fable scored 33%, while on the full challenge it landed closer to 14-15% range
The part II post linked in my top comment has a list of the challenges I used for that.
To run them, just point your model at the paper, it can find the repo, etc and with a key or sub, get it running
Thanks, that's helpful. Do you still have the full data of pass/fail for each test and model? It would be interesting retrospectively to see if there is such a subset.
I built some tools for personal research projects that could potentially help success rates on long dev tasks, empirically it seems to but I've never actually done a quantitative evaluation. SlopCodeBench looks promising for all the reasons you point out in your post.
Great benchmark!
I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.
yep agree
Suggestion to include GPT 6.1 Sol (Low) since that is configured as the default. I’d be interested to see how well it fares against the others.
I can’t find this useful given they are trying to quantify wether the code ‘feels’ sloppy with some fuzzy math that can easily be manipulated. at least it’s an attempt, some feel it’s a good attempt, i feel it’s rather poor and misses the point. the best code is the one that's most easily understood and equally performant - some of the math here may go against that (given you can reduce concentrations with helper functions and the like)
First of all, what happened to Max and Ultra?
Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?
Same here. I’m feeling productive enough. If I was more “productive”, I wouldn’t be able to keep up with what I was producing. My attention and cognitive broadband is the bottleneck. I find it hard to believe that people are actually productive using 4 active chat windows simultaneously for 8+ hours a day.
Same I just leave 6.1 on Max. Don't touch that dial
max and ultra are so slow and tend to overengineer massively imo
Max and ultra are slow and overengineer, based on evidence from evaluations.
For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning or if I want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.
IMO slop usually comes from overly defensive programming from LLMs. They’re loath to remove code and simplify, because unless we humans tell them some code / use case is unimportant, they have no reason to not care about it.
So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.
It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.
They also seem to really bias towards tacking on new code to "work around" what's there, rather than breaking down what's there into smaller more composable bits that can serve the pre-existing use case and the new requirement elegantly.
LLMs write. It sounds glib but it's what they do, and so every problem they encounter they will try to write their way out of it without lots of hand holding.
1 reply →
It's like we took the worst behavior of our colleagues, and went and said "yes, let's automate _that_ part" and patted ourselves on the back.
1 reply →
Yes exactly. And I’m not sure we want agents with that level of autonomy to decide what functionality is important and what to leave in place.
1 reply →
I have a simplify skill built from a rich hickey talk that basically solves the erosion and verbosity metrics SCB flagged (i.e. erosion and verbosity are way better and improve over time). However, after running the SCB benchmark I cannot find any evidence that it actually improves test pass rate trajectories. Food for thought on maintainability metrics.
Very interesting. If I’m honest with myself it could be very much a reality that I would be better off in only using for medium effor size tasked instead of trying to "offload" the necessary architecture thinking to "high" or even more.
They are good for inspiration though, but maybe I rather let me build a demo and then tell a medium model what I want from it.
wondering how gemini 4 performs
GPT 5.5 on Medium is all I use lately, I don't understand why anyone would pay drastically more for slightly more, when Opus has been good enough for a year now to do most things.
You unknowingly substituted the name of another model when you’re otherwise talking about the competitor fwiw. Some if those in the reply chain didn’t even seem to notice.
That’s a little telling of the state of ai atm though isn’t it? It’s all pretty similar. The latest Opus/GPT/Gemini/Deepseek/Kimi on medium does indeed do the job
It would be a little strange but not that strange for someone to have an idiosyncratic preference for GPT-5.5. It's still available as a Codex model (I think? It's selectable in Pi, not Codex itself), and it might perform differently enough from later GPT models that someone prefers it.
To your point, though, I don't know if I've noticed that big of an impact going from 5.5 to newer GPT models. If I had to switch back, I'm not sure how much I'd miss the newer ones. People talk about e.g. Opus 4.6 similarly, I've noticed.
The bigger change in OpenAI-land IMO is the models got cheaper, generally.
You're right, I meant to write: Opus for all of the above. I do think a lot of it is roughly similar, for most use cases, anything after Opus 4.5+ is "good enough" basically.
Why not GPT-{5.6, 6, 6.1} Sol? They're all cheaper than GPT 5.5, and as "smart", if not "smarter", at least to me.
Though, it has been a while since I've used 5.5.
Surely they mean Opus-5.5...
1 reply →
I have no data to support this, but when working on SaaS systems I find anthropic models more effective.
But for webdev, deep research, code review, and a few other tasks I find the current 6.1 sol better.
But I use both. And I use codex more.
This is interesting. Is nobody training LLMs to repeat resolving issues by patches from real projects where refactoring is required, judging the refactor similarity and not just tests passing?
what are the odds of as purely side effect of benchmaxing that the models get actually good because the benchmarkers the nittier they get the better the model has to become?
corrolary: just as cars evolved for worst because they were measured for "race numbers", what are the odds that the models just become "that uber nerd that degrades anyone who doesn't follow all his uber-rules for uber-code" and in the process start creating code that cannot be read or maintained?
> The full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep all tested behavior working as requirements change
So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons.
Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong.
I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (https://github.com/ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.
Related. Others?
Benchmarking Opus 5 on SlopCodeBench - https://news.ycombinator.com/item?id=49076391 - July 2026 (119 comments)
Wouldn't it be better to enforce pass via the harness and then measure cost/quality since that's more what people are likely to do?
Are the raw outputs published somewhere?
I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!
Finally a serious benchmark!
why measure something you can just enforce with something like opcore which we have built? https://github.com/the-open-engine/opcore/
Opus did better on medium? Can anyone confirm this somehow? What’s the sweetspot in your opinion?
https://www.reddit.com/r/LocalLLaMA/comments/1xatxqk/qwen38f...
Recent evaluation result.
I find QFN on xhigh usually overruns the context window for no good result. If it can do the task, it'll do it quicker on medium. If it can't do the task, xhigh won't change that.
I wish a company named human layer won't write their reports with so much Claudisms.
[flagged]
I think people downvoting you missed the sarcasm.
1 reply →
not suprising. i think we also tend to do better at things when we only expend medium effort.
The irony is the article is very sloppy. Can we at least write it and review it properly..
This is true. I was unfamiliar with this concept (I use codex/claude daily), and am trying to get to the "bottom line".
Oh, it's there at bottom under "my takeaway". But I still didn't get it initially.
Slop Code Bench? Need a Slop Writing Bench too
What's wrong with it? Reads fine to me.
There are 36 tests. You can pass 0 or 36 or any number inbetween. Lets give five significant digits of how accurate solving was.
That is not how data physics work. It should not be how data science works.
[flagged]
[flagged]
[dead]