Comment by TeMPOraL

I was thinking the next generation of attacks will involve playing the system prompt & harness the same way return-oriented programming does with normal binary code. That is, setting things up so that the system prompt itself implies the agent should do what the attacker wants them.

But that's probably off the table now that Anthropic is spreading the gospel of instruction minimization (which I assume but haven't checked applies to their own system prompt as well).

Source of this realization? Claude Code on the Web consistently tells me it wrote me some code on a branch but did not open a PR because I told it not to open PRs. Thing is, I never told it any such thing. There's some setting somewhere that's flipped, that activates a bit of its system prompt, and makes the agent not do what I asked it to do and claim I told it not to do it.

(It's probably trivial to find the offending switch, but I've been too lazy to do it yet.)