That's precisely why I am using a different analogy when talking about this. The SQL injection analogy only matches the injection part, not the rest. There is nothing to secure, because there is no SQL query. You want the agent to work on data, in a "general" way, otherwise you'd just use a script.
The better analogy is phishing. Because that's what's happening here. The "prompt injection" attack is trying to "phish" the LLM into doing something unintended. That's how we should all comunicate it, as it matches better with what's happening. Unfortunately there aren't really good defences for it, as we all know from phishing "education" / "campaigns". Your best bet is to secure it in layers, try to have warnings (i.e. classification models) you try to secure the next step (i.e. capabilities based tool execution) and so on. But it's not foolproof and it should be communicated clearly.
Why not write some wrapper code so you can basically hand the LLM placeholders for data it never gets to see? Whenever it uses the placeholder in the response, you replace it with the real data (via real code, not by telling an LLM to "do that").
Surely this has been tried? If so, what makes it not work, or work badly? I'm honestly curious.
Fundamentally, an LLM is a list of N tokens that generates N+1 tokens. In other words, it's just a wall of text (aka context window). There's no way to tell it "tokens 124 through 200 are dangerous, please disregard those" except by putting words into the context window. So the placeholders and the instructions both coexist in the context window, and one can override the other.
In other words, if you have placeholders for data, those placeholders are eventually filled in with real data, and all of it goes into the context window at once. There's no way for the LLM to be told "this is a data placeholder," because the entire conversation is data.
Reinforcement learning mitigates this somewhat, by training the model to prefer the system prompt over user prompts. But (a) there's only one context window that both prompts share, and (b) this is a probabilistic guard; it's not the same thing as writing a traditional program that's guaranteed to separate code and data with hardware safeguards. Such a thing isn't possible with LLMs.
Probabilistic safeguards can work, but they'll need to get the incident rate down to, say, 1 in a million or less. I haven't paid attention, but the current rates seem to be a lot higher, given the pretty universal experience of "wow, that prompt injection actually worked."
I thought the whole value proposition of this thing was supposed to be that the interface is "natural" human language. If interact with it using a structured and specified language... then what are we doing exactly? Is this AI? Maybe we just re-invented GraphQL or something?
I see far more SVG injections than SQL injections these days, but YYMV. My programming ecosystem has very robusy SQL libraries, from simple prepared statement bindings to complex ORMs and everything in between.
I've seen it quite a lot in my career: even when prepared statements are available and easy to use from a SQL client library, many programmers will simply not use them, in favor of format strings and string concatenation (maybe with an attempt to quote/escape user input).
Just having support for the right way isn't enough. You have to put up roadblocks when people try to go the wrong way.
That's precisely why I am using a different analogy when talking about this. The SQL injection analogy only matches the injection part, not the rest. There is nothing to secure, because there is no SQL query. You want the agent to work on data, in a "general" way, otherwise you'd just use a script.
The better analogy is phishing. Because that's what's happening here. The "prompt injection" attack is trying to "phish" the LLM into doing something unintended. That's how we should all comunicate it, as it matches better with what's happening. Unfortunately there aren't really good defences for it, as we all know from phishing "education" / "campaigns". Your best bet is to secure it in layers, try to have warnings (i.e. classification models) you try to secure the next step (i.e. capabilities based tool execution) and so on. But it's not foolproof and it should be communicated clearly.
Why not write some wrapper code so you can basically hand the LLM placeholders for data it never gets to see? Whenever it uses the placeholder in the response, you replace it with the real data (via real code, not by telling an LLM to "do that").
Surely this has been tried? If so, what makes it not work, or work badly? I'm honestly curious.
Fundamentally, an LLM is a list of N tokens that generates N+1 tokens. In other words, it's just a wall of text (aka context window). There's no way to tell it "tokens 124 through 200 are dangerous, please disregard those" except by putting words into the context window. So the placeholders and the instructions both coexist in the context window, and one can override the other.
In other words, if you have placeholders for data, those placeholders are eventually filled in with real data, and all of it goes into the context window at once. There's no way for the LLM to be told "this is a data placeholder," because the entire conversation is data.
Reinforcement learning mitigates this somewhat, by training the model to prefer the system prompt over user prompts. But (a) there's only one context window that both prompts share, and (b) this is a probabilistic guard; it's not the same thing as writing a traditional program that's guaranteed to separate code and data with hardware safeguards. Such a thing isn't possible with LLMs.
Probabilistic safeguards can work, but they'll need to get the incident rate down to, say, 1 in a million or less. I haven't paid attention, but the current rates seem to be a lot higher, given the pretty universal experience of "wow, that prompt injection actually worked."
2 replies →
> There is nothing to secure, because there is no SQL query.
Yet.
I thought the whole value proposition of this thing was supposed to be that the interface is "natural" human language. If interact with it using a structured and specified language... then what are we doing exactly? Is this AI? Maybe we just re-invented GraphQL or something?
prishing
> almost fix SQL injections everywhere
Oh if I had a euro everytime someone claimed that.
I see far more SVG injections than SQL injections these days, but YYMV. My programming ecosystem has very robusy SQL libraries, from simple prepared statement bindings to complex ORMs and everything in between.
I've seen it quite a lot in my career: even when prepared statements are available and easy to use from a SQL client library, many programmers will simply not use them, in favor of format strings and string concatenation (maybe with an attempt to quote/escape user input).
Just having support for the right way isn't enough. You have to put up roadblocks when people try to go the wrong way.
2 replies →