Comment by andai
1 day ago
LLMs regularly confuse my messages with their own. They'll say "you said xyz" when it was actually them. I heard the same principle can be used to bypass security restrictions, so maybe it's still an issue. (Or do they only get confused about authorship when security isn't at stake?)
I have literally never seen this and it doesn't make much since because the turns are flagged with metadata. It should be impossible for Claude to confuse your text for its own. I know because I use this metadata all the time. I've built hooks and processes against it.
Contexts get compacted, texts get copy-pasted, tasks get transferred between agents, humans assume ownership of LLM generated content without reading it, etc. etc.
Yes all user error and avoidable using good processes but I have not seen any organization care about this so far. The carelessness of AI adoption is crazy. People are not aware of the risks.