Comment by sddsfsdfsd2

It's a tricky problem for sure. Even on CPUs this separation is maintained by architectural guardrails. The CPU will happily execute whatever it is permitted to fetch. There is and cannot be a fundamental divide betwixt the two. It's always going to be an artificial externally managed issue. I suppose this is no different for LLMs.

My thinking is we are in the 50s/60s. Stuff is starting to come forward, it's all very exciting but very, very raw. I don't think this will last.

The notions of "tokens" and how inference works will become arcane insider knowledge like how CPU registers and interrupts work. You don't work with CPUs, you work with "computers" and even then mostly "operating systems" or even "browsers". Reality has been abstracted away from you to a very impressive degree. I don't think it'll be different here, but we haven't had our Xerox PARC and Bell Labs moments yet.