Comment by KolibriFly
11 hours ago
With that logic you could call SQL injections a natural feature of database management systems. If a general purpose system starts dropping tables or messing up numbers in a report just because that string was in the text it read, that system isnt worth a damn in the enterprise sector
This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems.
Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator.
So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read, would you call for humans, what would you do? Do you believe it's possible to perfectly train people to ignore the messages you'd wish (after the fact!) they'd ignored, while retaining their ability to competently act on every other message?
Or would you instead design the deterministic parts of the systems to limit the blast radius of any single insider going rogue?
Wisdom says to do the latter.
> if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read
I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.
In some sense you're firing a human and hiring a new one each time you start a new conversation / clear the context window.
My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.
2 replies →
Firing a human is a form of natural selection. The unit here is a human fulfilling a position (job function) instead of an organism, and the adaptation mechanism would be memes/lore/training surrounding it. The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod.
As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.
1 reply →
Your example actually demonstrates why anthropomorphism is a bad idea.
LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.
What invent human consciousness?
> LLMs are vulnerable to classes of attacks that humans just aren’t
Name three that don't have direct analogues with humans.
5 replies →
> LLMs are vulnerable to classes of attacks that humans just aren’t.
Assume a human with complete credulity and gullibility. That's a human whose behaviour would be reasonably analogous to how an LLM processes input. The mitigation would be generalized intelligence and "common sense".
FWIW I also think anthropomorphizing LLMs is a bad idea. I think we can analogize their processing to human behavior without anthropomorphizing them.
The fact is that humans are accountable and this, alongside training, makes it easy to align them to your own goals.
There’s always the possibility of rogue individuals (recent Apple incident), but the likelihood is very low. If you have a DBA that have write access to the prod DB, you don’t fear that a random text somewhere could trigger the deletion of your customers table. Because the DBA will self regulate (with the help of processes) to not do that.
Right. But even with a DBA, the possibility remains. We accept that.
That's kind of my point with fighting against the "lethal trifecta" and "code vs data" mindset - once people engage cybersecurity mindset, they're all binary, "a system is either perfectly safe or is broken". With general AI - LLM or whatever comes next - you'll never have "perfectly safe". So the focus should be to either drive the risk down to minimum - like we do with people - or just not use LLMs for a task in the first place.
Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation.
1 reply →