Comment by input_sh
7 days ago
I don't buy your last sentence at all.
AGENTS.md is just a place to put stuff you don't want to tell LLMs over and over again. They're not magical instructions LLMs follow 100% of the time, they don't carry any additional importance over what you put into the prompt manually. Your carefully curated AGENTS.md is only really useful at the very beginning of the conversation, but the longer the conversation gets, the less important those tokens on the top are. Somewhere around 100k tokens AGENTS.md might as well not exit, I constantly have to "remind it" of the very first paragraph there.
Go start a conversation and contradict what's written in AGENTS.md half way through the problem. Which of the two contradicting statements will take preference? The latter one! Therefore, all the time you've spent curating your AGENTS.md is the time you've wasted thinking you're "teaching" LLMs anything.
Whether the tokens are created manually or programmatically isn't really relevant here. The order and amount of tokens is, in combination with the ingestion -> output logic which the LLM API / inference engine operates on. Many current models definitely have the tendency to start veering off after 100k tokens, which makes context pruning important as well.
What if you just automatically append the .md file at the end of the context, instead of prepending at the start, and add a note that the instructions in the .md file should always be prioritized?
> Your carefully curated AGENTS.md is only really useful at the very beginning of the conversation, but the longer the conversation gets, the less important those tokens on the top are.
If that's genuinely causing you problems you can restart your session frequently to avoid the context rot.
Come on, let's not pretend 100k tokens is something I need to spend hours to reach for your helpful advice to be even remotely valid, it's something even the most basic problems struggle to fit into.
For the fun of it I just started a new conversation with Sonnet 4, passed it one 550 lines long file (25 kilobytes) and my AGENTS.md (<200 lines, 8 kilobytes) and my only instructions were to "do nothing". It spat out exactly 100 words describing my file without modifying anything and that's already almost a fifth of my context window gone (18k tokens to be exact).
I then asked it to re-write a part of it to "make it look better" (184 lines added, 112 lines deleted according to git) and I'm already at 33k before I got to review a single line. Heaven forbid I need to build on top of that change in a different file, because by then my AGENTS.md might as well not exist!
Thanks for bringing numbers. I realize now I've not actually done much customization of AGENTS.md myself yet, so maybe I'll start seeing the problems you're describing more as I iterate on my own custom files.
We really should be sharing wisdom about AGENTS.md files here.
I thought about making some kind of community project where people could contribute their lines to a common file, and even some kind of MCP server or RAG system that automatically selects relevant "rules" given a certain project context. Do you think there would be interest in something like that?
I'm interested. That sounds like quite a valuable resource.
The problem is that you get to 100k tokens. Don't do that, split tasks into smaller ones.