Comment by prometheus1992
7 hours ago
Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.
Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly use claude (opus) on typescript react projects that are well setup with minimal plugins/MCPs!
happy to share more if you are interested.
> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes:
https://lucumr.pocoo.org/2026/9/7/astra-why/
Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation.
It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.
In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)
The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"
One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't.
But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should.
I don't have empirical evidence to support that supposition though.
1 reply →
This is why I like to really interrogate Claude on what it’s planning to do, what the implementation will look like etc before writing code. Without enough direction and time thinking about it, it can definitely go in an unintended direction
The reasons are highly project specific. The closer your project is to CRUD, the higher the chance of success.
yes ... 'the closer it is to normative, the higher the chances of success'.
If it works but the design is terrible - is that still success?
In the short term yes. Long term, no. But I guess you can say the same thing about a team of engineers taking shortcuts
the requests went into a pipeline that turns them into de-identified, but salient, training data, yeah. everywhere except maybe bedrock.