Comment by gwerbin

LLMs are such fascinating things. So much emergent behavior and we probably still only understand a tiny amount of it.

I would love to know what was going on in the latent space that led to this particular sequence having a high probability in the token distribution. Was it something in the random gibberish? These things are really good at role playing after all, maybe the word "childish" shifted everything over into a latent role-playing space, like when something that makes no sense happens in a dream, but instead of your brain ending the dream it tries to roll with it and you end up with some really bizarre stuff happening.

LLMs often remind me of my intense autoimmune dreams from before I got diagnosed with celiac, not necessarily in content, just in how "autoregressive" they are, even our biggest and best models. I think that contributes to a lot of their uncanniness, an infinite stream of simulated consciousness that wanders indefinitely with no grounding conceptual center, like music with no stable total center and no consistent melody or structure (which is exactly what a lot of early AI generated music sounded like). I feel like only the latest generation of frontier models are powerful enough to ground the token distribution in a consistent topic over a long transcript, and even then an undisciplined user can essentially bait the model into following any number of side quests.