Comment by M0r13n
12 hours ago
This weekend I had some time with Opus 5.5 and GLM 5.3. I went over some findings in a library of Claude Mythos 5.1. These were minor, low-severity bug fixes and Mythos provided a PR to address them. Its PR was functionally correct. I still spent ~5 hours looping over them until I was really happy, even though a fully working PR already existed.
Some of the differences were:
- Mythos recommended salvaging damaged labels. I pushed back: data whose structure is broken shouldn’t be trusted
- My simpler more readable test caught a regression that Mythos’s more elaborate test missed
- Mythos allowed invalid values, citing real-world data that I could not reproduce, while I decided to follow the spec
- Mythos kept the old error types. I made every kind of malformed input raise the same error, and accepted a small, documented breaking change for anyone who relied on the old ones
Overall, I much preferred the final version of my PR to the initial draft Mythos provided. However, getting there involved quite a bit of time messing around with the code, as well as some back and forth with Opus (which was valuable).
This is a recent example that matches my overall experience working with current SOTA models (the ones mere mortals like me have access to).
TL;DR: I don’t think AI already has better judgment than (most) senior people. Obviously, that might change with future models. But that’s a hypothetical scenario. It might as well not change anytime soon.
Resonates exactly with my experience.
I'll add they also seem to make 'bad' judgement calls around which layer to implement something, often preferring downstream fixes/hacks/workarounds when upstream fixes would be better.
They also duplicate code lots rather than finding and reusing existing.
Both of these patterns lead to degradation and time wasting in complex code bases. Part of the behavioural adaptation I've made in using these tools is heading these off as much as possible at prompt-time
My prompts are often verbose to several paragraphs
> often preferring downstream fixes/hacks/workarounds when upstream fixes would be better.
It sometimes feels like the model is too lazy to do the actual reasonable fix and implements a _hacky_ fix downstream. And to be honest: I feel that!
This is true, but we also don’t know the long term architectural tradeoffs that are right for an LLM-maintained codebase.
For human-maintained code you manage duplication and locality of changes in the codebase because you want concepts to fit inside a human’s working memory. Keeping method bodies short is so they fit in one code window on the screen.
LLMs’ working memory works in very different ways; they see codebases not as windows of code but as blips of grep results. It might be that it’s easier for an LLM to just ensure five pieces of code are all modified in sync with one another than to safely implement a DRY refactoring.
4 replies →
That mostly matches my experience, but there is a caveat that you're not considering.
You don't know what good judgement calls the AI made along the way. Or rather you do but not in detail because those are the part you looked at that said, okay, this is good code and I should ignore it. Your judgment is biased because you're looking at the bad parts and not the good parts.
I think we are already at the point where code quality doesn't matter anymore because no human can keep up with the output. I don't like the code that these AI agents produce, but what do I care when I no longer have to read the code? As long as other AI agents can continue to understand it, it's good enough.