Comment by buildbot
6 hours ago
I’ve observed exactly these patterns with Opus and Fable as well - for example, forgetting that they can edit files and instead use python scripts as a patching tool…
6 hours ago
I’ve observed exactly these patterns with Opus and Fable as well - for example, forgetting that they can edit files and instead use python scripts as a patching tool…
The harness instructs them to behave this way. Also this approach saves tokens. The scripts allow to edit files in bulk, and most of the session cost is in cache reads (e.g. for 300K context each command costs the same as 30K input tokens).
> The harness instructs them to behave this way. Also this approach saves tokens. The scripts allow to edit files in bulk, and most of the session cost is in cache reads (e.g. for 300K context each command costs the same as 30K input tokens).
I understand the reasoning, but at that point wouldn't the LLM be better off creating `sed` commands and executing those? I mean, if it's already executing Python, it can literally do anything to the environment, so using `sed` is at least as safe, with a bonus that it (or a subagent, or a human) can double-check the intention with the sed script and flag incorrect or missing changes.
I've experimented quite a bit with giving agents python vs sed + awk. They make mistakes with both, a lot. The only thing that has stood out is that agents reach for python too quickly if it's available, and that awk causes the least problems, while sed might take several attempts to get results, similar to python.
Also it's the only way that makes sense when you need to work with big files, or large amount of files, or documents that look small when fetched through a RAG tool, but then you read one and get hit with couple megabytes of base64-encoded binary data you didn't expect because RAG tool stripped out embedded images...
Ask me how I know. Or don't. I have a standing rule for all agents warning about that failure mode (and related, doing `ls` in `/tmp` and few other directories that like to accumulate files by the hundreds..)
My harness forbids it, they end up spending time debugging their scripts
Why would you use a constrained edit tool when you are also allowed to use the complete power of python?
Because the complete power of Python also includes the power to fuck things up.
So does using an LLM.
2 replies →
Having an agent edit 100 files means the job will definitely get done correctly. When it writes a script to bulk edit things it fucks up and spends ages debugging their script.
Have you ever counted the number of times Claude fucked up quoting/escaping and had to issue a corrected tool call? Or get stuck in some tricky quoting situation for two minutes, throwing a couple piles of shit at the wall to see what sticks. IIRC I’ve even seen it eventually using the edit tool out of frustration once.
Simple is better than complex Complex is better than complicated
Or something, I don't remember...
... simply the best, better than all the rest (Tina Turner)
1 reply →
That was pre LLM. Now everything is a prompt that you type into AI lol
Why even offer the edit tool in that case? Also, what kind of editing could they possible do what wouldn't be possible with POSIX ed?
You can chain a lot more commands together with this technique than with a single Edit tool call.
1 reply →
This is an instruction by the harness. It re-injects the prompt every other message, so that's why it "forgets" to use the Edit tool.
this is intentional, afaik agents do better with python and alike than the harness tooling.
Using python or any other stone-age approach for search and replace is stupid when your language provides you with a complete, fully typed AST, like .NET does.
I use AST replacers, much more reliable.
Sounds like you don't have enough experience with coding agents. Deterministic scripts must always be preferred instead of LLM tool calls. In fact, you should instruct your agents to write code to execute instead of letting them call tools.
Plenty of experience ;)
Then why did you comment what you commented, good sir/madam. Claude and Codex are good at remembering to use scripts instead of tools these days, especially if your <32kb .md file mentions it. Not even talking about the skills designed to catch such issues.
Sounds like you completely lack all reading comprehension ability
LLMs sometimes like to execute one-off Python scripts to make edits to files rather than just calling the edit tool directly. Both are tool calls so saying that you should have it write code instead of doing tool calls makes no sense because writing code is a tool call for it...
[flagged]
9 replies →