Comment by z3c0

There is plenty more complexity, but that emerges more from embedding, where the less superficial elements of information (such as syntactic dependencies) allow the model to hone in on the higher-order logic of language.

e.g. when preparing the corpus, embedding documents and subsequently duplicating some with a vec where the tokens are swapped with their hex repr could allow an LLM to learn "speak hex", as well as intersperse the hex with the other languages it "knows". We would see a bunch of encoded text, but the LLM would be generating based on the syntactic structure of the current context.