Comment by qlte
3 hours ago
> People ... are trying very hard to argue that humans are subject to this via social engineering but it is not the same
Thank you, I always hear the "but humans fall for social engineering too!" line used reflexively whenever yet another prompt injection attack gets reported and it drives me crazy. While it's true certain strings of text exist that both an LLM and a human could plausibly fall victim to, they are a tiny fraction of the nearly unlimited permutations of text that are complete gibberish or invisible for any human but parsed instantly (and dangerously) by an LLM.
Base64, Unicode substitution, emojis, output of obfuscated but "harmless" code run in a sandbox, image steganography, etc that could be endlessly disguised without a human even being able to see it, yet alone fall for it. The attack surface is massively expanded for an LLM agent vs. a gullible Tier 1 customer service worker.
> Base64, Unicode substitution, emojis, output of obfuscated but "harmless" code run in a sandbox, image steganography, etc that could be endlessly disguised without a human even being able to see it, yet alone fall for it
Like a whisper or a morse code pattern or a post-it stuck in the middle of a stack of fresh printouts saying "${employee} is threatening to kill me please call 911" or...
Yes, LLMs and humans have different sensory inputs. That's immaterial; the "problem" isn't in the intersection of LLM and human sensoria, but in what happens once those inputs reach the model/brain.
> I always hear the "but humans fall for social engineering too!" line used reflexively whenever yet another prompt injection attack gets reported and it drives me crazy.
Weird, because all I see over this site is people reflexively claiming that prompt injection is like SQL injection and "lethal trifecta" is a software engineering problem.
It is not. Prompt injection is, fundamentally, the same class of thing as social engineering. Treating LLMs as regular software is a fundamental category error. Until people finally get that, we won't get much progress in this space, because all the effort gets wasted on barking up the wrong tree.
Agreed. I do not understand why so many others cannot understand that independent non deterministic models cannot be constrained the same way as deterministic code can.
It’s just fundamentally different. They are both software in the same way liquids and solids are both matter but they have fundamentally different properties due to their nature.