Comment by __MatrixMan__

While I use LLMs I form and discard mental models for how they work. I've read about how they work, but I'm looking for a feeling that I can't really get by reading, I have to do my own little exploration. My current (surely flawed) model has to do with the distinction between topology and geometry. A human mind has a better grasp of topology, if you tell them to draw a single triangle on the surfaces of two spheres they'll quickly object. But an LLM lacks that topological sense, so they'll just try really hard without acknowledging the impossibility of the task.

One thing I like about this one is that it's consistent with the Waluigi effect (which I just learned of). The LLM is a thing of directions and distances, of vectors. If you shape the space to make a certain vector especially likely, then you've also shaped that space to make its additive inverse likely as well. To get away from it we're going to have to abandon vector spaces for something more exotic.