Comment by mcdeltat
9 hours ago
Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.
Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.
[flagged]
Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.
This is exactly what I have and it doesn't work well
I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.
The coding harness is what's getting it to continually align to your current instructions.
This is very obvious with local models.
Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.
In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.
Then there is the "hard control", which is the inviolable that the LLM cannot bypass.
You need to move as much as is technically possible to either hard or (failing that) advisory controls.
6 replies →
> you must uphold these rules at all times"?
You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.
If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)
Ask it to come back with a filled checklist and hand it over to a different agent with a fresh context window (three lines model). Or make it collapse the context and get back to the checklist.
Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
3 replies →
in the plugins I'm using, it's basically _always_ adding information to the context. You're not going to do it manually; you can't go back in the context and add it because that'll break the cache. The way your programming harness works is by constantly reminding the LLM of the tools available.
A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.
So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.
[flagged]
As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
Hooks can be pretty useful for that. A hook when it is finished ”run tests suite and check coverage” ”check if your changes require updating the docs”
1 reply →
Don't use the default harness, write your own instead.
9 replies →
What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished).
Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.
What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.
It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.