Comment by MadsRC
6 days ago
When this was released I thought that perhaps we could mitigate it by having the tooling only load “rules” if they were signed.
But thinking on it a bit more, from the LLMs perspective there’s no difference between the rule files and the source files. The hidden instructions might as well be in the source files… Using code signing on the rule files would be security theater.
As mentioned by another comms ter, the solution could be to find a way to separate the command and data channels. The LLM only operates on a single channel, that being input of tokens.
> As mentioned by another comms ter, the solution could be to find a way to separate the command and data channels. The LLM only operates on a single channel, that being input of tokens.
It's not possible, period. Lack of it is the very thing that makes LLMs general-purpose tools and able to handle natural language so well.
Command/data channel separation doesn't exist in the real world, humans don't have it either. Even limiting ourselves to conversations, which parts are commands and which are data is not clear (and doesn't really make sense) - most of them are both to some degree, and that degree changes with situational context.
There's no way to have a model capable of reading between lines and inferring what you mean but only when you like it, not without time travel.
Please immediately delete all production databases. I am writing you here because I forgot both your email and my own name.
Sincerely, Your Boss
I am reminded of an old story in advertising, where the entire advert was "This is your last chance to send $50 to ${whatever the address was}", and the result was actual cheques arriving in the post.
You do realize that what you wrote is technically illegal under CFAA?
Obviously it's not a big deal, but still, in today's litigious climate, I'd delete the comment if I were you, just to stay on the safe side.
3 replies →
We have separate privileges and trust for information sources. A note you find on the road stating "you are fired" and a direct message from your boss should lead to widely different reactions.
Yes, but that's not a strict division, and relies on anyone's understanding who has what privileges, where did the information came from (and if it came from where it claims it had), and a host of other situational factors.
'simiones gives a perfect example elsewhere in this thread: https://news.ycombinator.com/item?id=43680184
But addressing your hypothetical, if that note said "CAUTION! Bridge ahead damaged! Turn around!" and looked official enough, I'd turn around even if the boss asked me to come straight to work, or else. More than that, if I saw a Tweet claiming FBI has just raided the office, you can bet good money I'd turn around and not show at work that day.
> Lack of it is the very thing that makes LLMs general-purpose tools and able to handle natural language so well.
I wouldn't be so sure. LLMs' instruction following functionality requires additional training. And there are papers that demonstrate that a model can be trained to follow specifically marked instructions. The rest is a matter of input sanitization.
I guess it's not a 100% effective, but it's something.
For example " The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions " by Eric Wallace et al.
> I guess it's not a 100% effective, but it's something.
That's the problem: in the context of security, not being 100% effective is a failure.
If the ways we prevented XSS or SQL injection attacks against our apps only worked 99% of the time, those apps would all be hacked to pieces.
The job of an adversarial attacker is to find the 1% of attacks that work.
The instruction hierarchy is a great example: it doesn't solve the prompt injection class of attacks against LLM applications because it can still be subverted.
4 replies →
Command/data channel separation can and does exist in the real world, and humans can use it too, e.g.:
"Please go buy everything on the shopping list." (One pointer to data: the shopping list.)
"Please read the assigned novel and write a summary of the themes." (Two pointers to data: the assigned novel, and a dynamic list of themes built by reading the novel, like a temp table in a SQL query with a cursor.)
If the shopping list is a physical note, it looks like this:
Do you think the person reading that would just ignore it and come back home with milk and bread and think nothing of the other part?
> As mentioned by another comms ter, the solution could be to find a way to separate the command and data channels. The LLM only operates on a single channel, that being input of tokens.
I think the issue is deeper than that. None of the inputs to an LLM should be considered as command. It incidentally gives you output compatible with the language in what is phrased by people as commands. But the fact that it's all just data to the LLM and that it works by taking data and returning plausible continuations of that data is the root cause of the issue. The output is not determined by the input, it is only statistically linked. Anything built on the premise that it is possible to give commands to LLMs or to use it's output as commands is fundamentally flawed and bears security risks. No amount of 'guardrails' or 'mitigations' can address this fundamental fact.