Comment by skinfaxi
2 hours ago
Yeah but unless someone has my private key they can't sign messages as me. Why can't a harness deny tool use (or offer a limited set of tools) to any agent consuming unsigned content?
2 hours ago
Yeah but unless someone has my private key they can't sign messages as me. Why can't a harness deny tool use (or offer a limited set of tools) to any agent consuming unsigned content?
1. The LLM is passed a context window
2. The LLM outputs text containing "call tool X"
3. The tool outputs text
4. The LLM is passed another context window, containing the original text plus the tool call output
How has the signature helped matters? If you discarded the malicious text in the first place, there was no need for it be unsigned/unverified. If you didn't discard it, it's still present in the LLM input at step 4, there for the LLM to """""mistakenly""""" use to shape its output.
Replying a different way: if you made it so the LLM were not, under any circumstances, allowed to do harmful things, then it would be harmless. The reason it's not harmless is that people want the LLMs to be able to do things like deleting a document... but they only want the LLM to do that when they intended for it to do so, and the only arbiter of their intent is their own text which is arbitrarily mixed in with third-party content.
For example, users want to be able to say "email the document referenced in this email to Claire". So the LLM needs access to functions read-this-email(), search-document-by-name(), and email-designated-user(), and there's your problem.