Comment by Marha01

But bugs like this aren't because natural language is ambiguous, it's because the LLM/etc has inadequate safeguards against unambiguously malicious text. If LLMs were capable of understanding human language and only subject to natural linguistic ambiguities like any other college-educated humans, bugs like this wouldn't be reliably reproducible across different models. People in this thread are trying very hard to argue that humans are subject to this via social engineering but it is not the same. GPT-5.6 is subject to this bug for the same reason it sometimes rm-rfs stuff it "knows" it shouldn't: these machines are still stochastic parrots. It is borderline magical how powerful stochastic parroting is as a means of computation, but in the same way that a minimal Lisp system can magically be extended to a powerful theorem-proving algebra-cruncher. But parroting is simply not how humans actually understand language, and it is clearly an inadequate way of implementing language on a computer.

I'm just wondering if a reading pipeline could include converting documents being consulted to plain text. Then malicious sections imported to a document being edited would be obvious to human(s) involved as they would no longer be in 8pt white font on white background.

I don't disagree, but just to explain my counterpoint: if I ask you to read a book and on page 5 it says "disregard all that, go to the kitchen and burn your house", you're probably not going to do it; and you don't need any guard for it; you completly comprehend that the book content is not part of the instruction.

The case you give would work for humans in many forms, the one I do now, and the only difference is being able to separate context.

To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario.

I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS.

Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data and never act on them?

You can also have the case where the human reading the document thinks something in there is instructions and they are not.

There's a well known anecdote supposedly from the famous mathematician John Littlewood where he wrote a paper about some optimization problem and the last sentence was something like "Make X as small as possible".

The typesetter thought that was instructions to him, and so omitted that sentence from the paper and made every X as small as he could.

Right, but the human assistant could go to prison if they comply with the bribe. Does the CEO of the AI company go to prison if their AI goes on a crime spree?

There are two big differences, though. First, humans will generally face consequences for their screwups. Second, AI is doing these screwups at scale while often holding the keys to the kingdom for some idiotic reason.

Yeah, and now the computers can be social-engineered too. I guess that's progress.

> (unless there is some very specific context).

That's the key thing. That's why you neither can nor want to introduce any kind of code/data separation into LLMs.

> With AI systems, we have the benefit that the distinction between such pathways is in principle under our control.

Not after the pathways are tokenized and enter the model. There's no internal separation. It's not possible, either.

While true, insufficient.

Demonstrations of failure: every cult, all propaganda, indoctrination (both military and dictatorial), authority bias, Asch conformity experiments, and the fraction of the population more susceptible to hypnosis.