Comment by overgard
3 months ago
In my case, where I see it most often is when the LLM has to rework something multiple times, and the feedback loop is vague (especially when all I have to give it is "no error messages, but it's still broken"). It seems like after the third or fourth try it just kinda goes off the rails. I find that the one-shot quality tends to be a little better, if the slot machine happened to work correctly that time.
You shouldn't be using an LLM directly (web chat style). A proper harness allows an agent to see the errors itself and correct as needed. You can the correct it at higher, more meaningful levels.
My experience is in doing this with Claude/Codex/OpenCode with a pretty rigorous setup (AGENTS.md/CLAUDE.md for specific subfolder rules, strict compile/test/lint rules. This isn't me copy-pasting from web chat.
I've been building a general linter tool to help keep repos in a consistent and clean shape when working with AI [0]. You can define repo and file layout, structure, hygiene rules and have them checked in pre-commit, in ci, or manually. It also integrates and plays well with AGENTS.md - allowing exporting agent instructions from the alint rule config [1].
[0] https://github.com/asamarts/alint
[1] https://alint.org/agent-friendly-linter/