Comment by existencebox
1 day ago
I expect there's a probabilistic factor to it, among other things.
I use the almost the same addendum as you ("do not make, commit, or push any changes without my approval") and it broadly seems to work in my day-to-day dev. However, I also set up some agents that trigger on incoming requests from my dev teams to automate things that used to be rote/manual tasks. _That_ is where I see the issues, I suspect simply because it has so many more opportunities to go off script.
The most common failure mode is that it will, despite all directives to not make changes without approval, start writing unit tests. (I do find this equal parts head-shaking and amusing; At least it chooses something relatively 'benevolent' vs. the aforementioned skynet.)
I haven’t used these tools that permit LLMs to make changes on their own, but I can’t help being baffled if there’s no big red switch that changes the system prompt and disables any and all write access to the outside world.
There is - in the Claude Code harness there's a dedicated Plan Mode that doesn't edit any files, or a mode where you can manually approve each edit.
If you're running Claude via -p automated mode, you can set all kinds of permissions and access to each individual editing tool, much like you would with firewall software.
It doesn't edit the system prompt, it blocks access to the tools itself. Just make sure you haven't given the AI the ability to edit its own configuration files, or you'll potentially have an OpenAI v Hugging Face situation.
I think the best way is to run agents under Unix user that has very limited rights (eg only read only) for project folder and the rest of the system. It’s kind terrifying how people run agents on their computers without any hard limitations, just hoping they will follow instructions and not do anything malicious. This is basically a trojan waiting to happen, either trying to solve something by all means possible, or getting prompt injection
I do wonder if "without my approval" is looser than "discuss only, do not make edits"? I'm not sure how much that finegrain thing matters in prompts nowadays tho
One problem is that agents are notoriously bad at handling ‘me’ vs ‘you’ over an extended conversation. ‘Without my approval’ far back in the context of an extended session gets fuzzy about who ‘me’ is.
This has always seemed strange to me. It will offer a suggestion, I'll make a comment about a suggestion, and it returns with, "Your suggestion..." The further distance from the suggestion, the more likely it confuses who made it.
2 replies →