Security minded programmers understand that. "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea.
And AI chatbots are very much targeted at the second group, not the first.
> "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea.
Because it's not a concept in the real world. Physical reality has no such separation, and neither do human minds.
Tell people you're discussing a board game or some sport, then they'll understand - other than bureaucracy (scary!) and school (traumatic!), that's the one kind of artificial system with rules affording for code/data separation that general population has most experience dealing with.
Genuinely curious, does the telephone system have enough scope to make it dangerous?
Elevators are extremely hackable all over the world. It’s generally not considered a problem because it requires physical access, specific knowledge, and defeating cameras to exploit successfully.
Some physical constraints do not allow for the best security, and those physical constraints will always win out in the real world. When presented with the pick two of three options of fast, cheap, secure/done right, fast and cheap will always win out.
Meta adding an AI chat window in whatsapp and Microsoft adding copilot in every word document was not done with developers in mind, which is why they're missing a lot of power user features that they'd surely have if they were targeted at developers.
You're mistaking the majority of what you see (like Claude et al) with the majority of stuff that is out there. The vast majority of ChatGPT, CoPilot and Gemini users are not developers and will never be.
When working on PDFKit for MacOS, one short-coming our implementation had was the lack of support for Javascript in PDF's.
Oops.
(I mean, I'm one engineer and I was not going to try and hoist a JS runtime in my little PDFKit framework. And besides, the sample PDF's we were running into with JS were rare—usually tax-like forms that would add numbers from A and B and display the result in C. It seemed like a huge effort for such a small gain . Oh, and a security vulnerability.)
People understand that. They just don't know how to implement that with LLMs
In the GPT-2 era LLMs were just data. Instructions did not exist, and if you added them to your data they would not be followed. Then around 2022 we figured out how to patch in instruction following with a bit of fine tuning, leading to the current AI bubble. That's an ugly hack that leads to all these issues. But it's what this entire AI bubble is founded on. And nobody seems to have found a better way (or at least one that actually scales and doesn't make unreasonable sacrifices)
Sure they would be. But for those old models, you'd have to prompt it in a framing of a screenplay or something.
You're forgetting that LLMs just output a stream of tokens - the interpreter that acts on those is a piece of classical code, and sits outside of the model.
> the interpreter that acts on those is a piece of classical code, and sits outside of the model.
Correct, but it's an LLM that's reasoning about what stream of interpretable tokens should be emitted. The interpreter can certainly apply some security measures around what's being asked of it (like ask for confirmation), but that can only go so far. Is the human in the loop always capable of understanding what's safe to execute? If not, should we pass it through another fallible LLM to help make that judgement call?
Some security measures can be handled in a purely deterministic manner. But not all of them, and that's the problem.
Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug.
People get too hung up on this fundamentally wrong idea, and the space of security, instead of progressing, is just running in circles like a headless chicken, making a mess of everything.
Literally all of software is artificial? Being explicit and reasoned about how you choose to allow or deny a particular computation is, surely, at the heart of a lot of computer security?
Code/data separation is at the heart of computer security in the same way slapstick comedy is at the heart of humor.
There's an endless supply of people who think they know what is Code and what is Data, and they're always arguing with others who also think that, and neither realize that Code/Data classification is an opinion, a perspective. It doesn't hold in general.
Having a separation like this makes sense for super narrow systems, where you can define the allowed and disallowed use cases, enforce the distinction (because it's not real - therefore you have to enforce it mechanistically within your system), and willing to accept that some useful operations will be denied by your system.
With that logic you could call SQL injections a natural feature of database management systems. If a general purpose system starts dropping tables or messing up numbers in a report just because that string was in the text it read, that system isnt worth a damn in the enterprise sector
This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems.
Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator.
So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read, would you call for humans, what would you do? Do you believe it's possible to perfectly train people to ignore the messages you'd wish (after the fact!) they'd ignored, while retaining their ability to competently act on every other message?
Or would you instead design the deterministic parts of the systems to limit the blast radius of any single insider going rogue?
> Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug.
Note: I'm parsing 'needs not to have them' as 'needs (not to have them)'. If you were using 'needs not' as an alternate for 'does not need' then never mind, although I'd guess that is not the case because the alternative for 'does not need' would be 'need not' rather than 'needs not' and you probably wouldn't make that mistake.
Doesn't this imply that it is not possible to implement a general purpose system on any of our current computing devices?
For all our current computing devices everything that can be done on devices that do not separate instructions and data can also be done on devices that do, and vice versa.
Different layers of abstraction. You can look at it this way: the machine separating instructions and data can still emulate a machine that doesn't. Within the inner machine, there is no such separation. Outside of it, but still within the outer machine, there is. The rules of the outer machine don't affect what's running in the inner one, but also what's running in the inner one can't affect the outer machine directly.
But I guess a different way of framing it is, what is "code" vs "data" for the machine is not the same as what we talk about discussing the LLM running in it. For the outer machine, all tokens are pure data.
A pure Harvard architecture machine has exactly that separation. Admittedly, there needs to be some mechanism for converting data to code so you can actually program it, but it doesn't have to be accessible by the device itself. E.g. programming the Microchip PIC16 series of microcontollers required driving the reset pin to 13V (enough to destroy any other pin). It's not possible without dedicated external hardware.
> A pure Harvard architecture machine has exactly that separation.
It emulates and enforces that separation. A mathematical abstraction of a Harvard architecture machine has that separation, the real machine merely emulates it, and is only able to do so within some specific constraints (such as: no one hooks up dedicated programmer to the chip, or no one undervolts or overheats the cheap in clever way, or no one takes a swing at it with an x-ray source, or...).
That's the other thing people forget here: we're emulating abstract mathematical universes with real atoms, and then we're stacking those abstractions within abstractions. There is a whole segment of computer security that deals with that. When we say "once attacker has physical access, it's game over", or even discuss "side channels", is when we briefly remember that computer systems live in physical world, and the rules of our carefully designed abstract universes don't hold when you're on the outside of them and reaching in.
If you hand me two sheets of paper, one of them containing instructions and another containing data, I'll have a pretty easy time keeping them separate, and I think most humans wouldn't struggle with that problem either.
> If you hand me two sheets of paper, one of them containing instructions and another containing data, I'll have a pretty easy time keeping them separate
You think. But there are ways around that. How about a credible extortion message targeting specifically you, that is embedded somewhere on the data sheet? Suddenly, the data has become the instructions...
There are so many better alternatives but it seems many people really like Word for some weird reason. The last time I cared I had to look up how to make a document starting the page numbering on the 2nd page. It turns out there are totally different ways between different versions of Word. shrug.jpg
I almost agree. Have you tried to add an image in LaTeX that does not wander to a random page?
\begin{figure}[HERE!!!!!!]
or something like that.
And in the old compiler, I remember a problem with bounding boxes, and keeping a eps and pdf version of each image to get a correct dvi and pdf. I think this part is fixed now.
What are the "so many better alternatives"? Google Docs is pretty decent but a fair bit more basic. Proper technical authoring systems like Typst, LyX and LaTeX are way too hard for the average person. LibreOffice is much worse than MS Word.
Tell that to middle bosses and CEOs and MS Office VBA bootlickers and Excel workshippers.
Meanwhile, CSV files parsed with custom reviewed AWK scripts can be 100% safe with charts made from Gnuplot. Heck, even some notebook like Ipython with a CSV module would be far more desirable than a spreadsheet. Any of them. Just look at the Genomics Disaster on Excel because of shitty parsing.
Security minded programmers understand that. "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea.
And AI chatbots are very much targeted at the second group, not the first.
> "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea.
Because it's not a concept in the real world. Physical reality has no such separation, and neither do human minds.
Tell people you're discussing a board game or some sport, then they'll understand - other than bureaucracy (scary!) and school (traumatic!), that's the one kind of artificial system with rules affording for code/data separation that general population has most experience dealing with.
Even engineers like doing it sometimes. The old telephone system was so hackable because of in band signaling.
Genuinely curious, does the telephone system have enough scope to make it dangerous?
Elevators are extremely hackable all over the world. It’s generally not considered a problem because it requires physical access, specific knowledge, and defeating cameras to exploit successfully.
What can you do with the telephone system?
3 replies →
Some physical constraints do not allow for the best security, and those physical constraints will always win out in the real world. When presented with the pick two of three options of fast, cheap, secure/done right, fast and cheap will always win out.
2 replies →
The “new” phone system (SS7) still relies on implicit trust and a lack of security.
3 replies →
> And AI chatbots are very much targeted at the second group, not the first.
I suppose this is why the AI labs are famously not releasing developer-oriented tools.
Meta adding an AI chat window in whatsapp and Microsoft adding copilot in every word document was not done with developers in mind, which is why they're missing a lot of power user features that they'd surely have if they were targeted at developers.
You're mistaking the majority of what you see (like Claude et al) with the majority of stuff that is out there. The vast majority of ChatGPT, CoPilot and Gemini users are not developers and will never be.
1 reply →
There's a (terrifyingly) large number of developers who don't qualify as "security-oriented programmers".
When working on PDFKit for MacOS, one short-coming our implementation had was the lack of support for Javascript in PDF's.
Oops.
(I mean, I'm one engineer and I was not going to try and hoist a JS runtime in my little PDFKit framework. And besides, the sample PDF's we were running into with JS were rare—usually tax-like forms that would add numbers from A and B and display the result in C. It seemed like a huge effort for such a small gain . Oh, and a security vulnerability.)
Arguably the biggest vuln of the filetype.
People understand that. They just don't know how to implement that with LLMs
In the GPT-2 era LLMs were just data. Instructions did not exist, and if you added them to your data they would not be followed. Then around 2022 we figured out how to patch in instruction following with a bit of fine tuning, leading to the current AI bubble. That's an ugly hack that leads to all these issues. But it's what this entire AI bubble is founded on. And nobody seems to have found a better way (or at least one that actually scales and doesn't make unreasonable sacrifices)
Sure they would be. But for those old models, you'd have to prompt it in a framing of a screenplay or something.
You're forgetting that LLMs just output a stream of tokens - the interpreter that acts on those is a piece of classical code, and sits outside of the model.
> the interpreter that acts on those is a piece of classical code, and sits outside of the model.
Correct, but it's an LLM that's reasoning about what stream of interpretable tokens should be emitted. The interpreter can certainly apply some security measures around what's being asked of it (like ask for confirmation), but that can only go so far. Is the human in the loop always capable of understanding what's safe to execute? If not, should we pass it through another fallible LLM to help make that judgement call?
Some security measures can be handled in a purely deterministic manner. But not all of them, and that's the problem.
1 reply →
Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug.
People get too hung up on this fundamentally wrong idea, and the space of security, instead of progressing, is just running in circles like a headless chicken, making a mess of everything.
Literally all of software is artificial? Being explicit and reasoned about how you choose to allow or deny a particular computation is, surely, at the heart of a lot of computer security?
Code/data separation is at the heart of computer security in the same way slapstick comedy is at the heart of humor.
There's an endless supply of people who think they know what is Code and what is Data, and they're always arguing with others who also think that, and neither realize that Code/Data classification is an opinion, a perspective. It doesn't hold in general.
Having a separation like this makes sense for super narrow systems, where you can define the allowed and disallowed use cases, enforce the distinction (because it's not real - therefore you have to enforce it mechanistically within your system), and willing to accept that some useful operations will be denied by your system.
2 replies →
With that logic you could call SQL injections a natural feature of database management systems. If a general purpose system starts dropping tables or messing up numbers in a report just because that string was in the text it read, that system isnt worth a damn in the enterprise sector
This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems.
Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator.
So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read, would you call for humans, what would you do? Do you believe it's possible to perfectly train people to ignore the messages you'd wish (after the fact!) they'd ignored, while retaining their ability to competently act on every other message?
Or would you instead design the deterministic parts of the systems to limit the blast radius of any single insider going rogue?
Wisdom says to do the latter.
17 replies →
> Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug.
Note: I'm parsing 'needs not to have them' as 'needs (not to have them)'. If you were using 'needs not' as an alternate for 'does not need' then never mind, although I'd guess that is not the case because the alternative for 'does not need' would be 'need not' rather than 'needs not' and you probably wouldn't make that mistake.
Doesn't this imply that it is not possible to implement a general purpose system on any of our current computing devices?
For all our current computing devices everything that can be done on devices that do not separate instructions and data can also be done on devices that do, and vice versa.
Different layers of abstraction. You can look at it this way: the machine separating instructions and data can still emulate a machine that doesn't. Within the inner machine, there is no such separation. Outside of it, but still within the outer machine, there is. The rules of the outer machine don't affect what's running in the inner one, but also what's running in the inner one can't affect the outer machine directly.
But I guess a different way of framing it is, what is "code" vs "data" for the machine is not the same as what we talk about discussing the LLM running in it. For the outer machine, all tokens are pure data.
A pure Harvard architecture machine has exactly that separation. Admittedly, there needs to be some mechanism for converting data to code so you can actually program it, but it doesn't have to be accessible by the device itself. E.g. programming the Microchip PIC16 series of microcontollers required driving the reset pin to 13V (enough to destroy any other pin). It's not possible without dedicated external hardware.
> A pure Harvard architecture machine has exactly that separation.
It emulates and enforces that separation. A mathematical abstraction of a Harvard architecture machine has that separation, the real machine merely emulates it, and is only able to do so within some specific constraints (such as: no one hooks up dedicated programmer to the chip, or no one undervolts or overheats the cheap in clever way, or no one takes a swing at it with an x-ray source, or...).
That's the other thing people forget here: we're emulating abstract mathematical universes with real atoms, and then we're stacking those abstractions within abstractions. There is a whole segment of computer security that deals with that. When we say "once attacker has physical access, it's game over", or even discuss "side channels", is when we briefly remember that computer systems live in physical world, and the rules of our carefully designed abstract universes don't hold when you're on the outside of them and reaching in.
2 replies →
Only in systems that need to be themselves super generalist. Which is almost never the case.
> Which is almost never the case.
Well, the topic is about AI..
LLMs are.
1 reply →
If you hand me two sheets of paper, one of them containing instructions and another containing data, I'll have a pretty easy time keeping them separate, and I think most humans wouldn't struggle with that problem either.
> If you hand me two sheets of paper, one of them containing instructions and another containing data, I'll have a pretty easy time keeping them separate
You think. But there are ways around that. How about a credible extortion message targeting specifically you, that is embedded somewhere on the data sheet? Suddenly, the data has become the instructions...
1 reply →
The code sheet says take some bits from the data sheet and interpret it as if it were on the code sheet.
There are so many better alternatives but it seems many people really like Word for some weird reason. The last time I cared I had to look up how to make a document starting the page numbering on the 2nd page. It turns out there are totally different ways between different versions of Word. shrug.jpg
Such as? Word hits the sweet spot of having support for all the complexity the average person may encounter/want to create.
Libre, Apple Pages, and Google Docs all seem like clearly worse tools in most aspects in my experience.
LaTeX is extremely powerful, but also way too complicated for the average non-HN person/person who doesn't live in complicated documents.
I almost agree. Have you tried to add an image in LaTeX that does not wander to a random page?
or something like that.
And in the old compiler, I remember a problem with bounding boxes, and keeping a eps and pdf version of each image to get a correct dvi and pdf. I think this part is fixed now.
What are the "so many better alternatives"? Google Docs is pretty decent but a fair bit more basic. Proper technical authoring systems like Typst, LyX and LaTeX are way too hard for the average person. LibreOffice is much worse than MS Word.
Code is data is symbolic reality. I don’t think people’s understanding changes this.
‘(Lisp would like a word)
And I thought people understood that.
The graybeards know it. But they only know it through experience. It's blue/red/pink box phone phreaking all over again.
The technology changes, but the mistakes remain the same.
People who use machines based on the von Neumann architecture?
Tell that to middle bosses and CEOs and MS Office VBA bootlickers and Excel workshippers.
Meanwhile, CSV files parsed with custom reviewed AWK scripts can be 100% safe with charts made from Gnuplot. Heck, even some notebook like Ipython with a CSV module would be far more desirable than a spreadsheet. Any of them. Just look at the Genomics Disaster on Excel because of shitty parsing.
Isn't React, the most popular JS library, an example of that? Clearly people don't understand that
No it’s not an example of that. Do you store components in your component state?
He probably means JSX mixing HTML with Javascript...
function Greeting({ name }) { return <h1>Hello, {name}</h1>; }
1 reply →