Comment by danlitt

> You can't guarantee an LLM does anything.

Agreed.

> But that doesn't mean that separation between instructions and data is impossible.

Yes it does! The comments you are replying to are concerned that it is not possible to be sure that data and instructions have been separated. With certain kinds of automated systems (traditional ones), unless you write them incorrectly, you can be sure of this. And it is possible to engage in a productive incremental process where mistakes can be identified and removed, in a way people comprehend and can plan around.

LLMs do not have this. They have heuristics and guesses. Nobody knows what will work ahead of time, nor even a probability that it will work. That is not a doomer comment by the way! The same is true when you talk to a person. But it is a fundamental limitation, it cannot be removed.