Comment by 0xbadcafebee

> The full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep all tested behavior working as requirements change

So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons.

Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong.

I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (https://github.com/ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.