Comment by 0xDEAFBEAD
9 hours ago
Why not just have distinct input streams, or a metadata stream which annotates text in the main stream according to priority in case of conflicting instructions?
9 hours ago
Why not just have distinct input streams, or a metadata stream which annotates text in the main stream according to priority in case of conflicting instructions?
Because nobody has figured out how to make that work 100% reliably yet.
The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#s...
Then you train your model to take those tokens into account.
Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277
> We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label