Comment by globnomulous

> because LLMs can only reason with written language

Is that right? Think of code that draws a square, the representation of that code in storage, the movement of electrons, and the 'actual' square on the screen -- these are all to us transcriptions of same thing across different domains or media, and we can deterministically translate back and forth between them, but there's no real, actual, essential identity property. The code isn't the square. The electricity isn't the code. The storage isn't the electricity.

I think of LLM reasoning and output similarly. The underlying graph of weights, matrices, and other data aren't knowledge, understanding, or language. 'Translating' the system's output to language is jusas valid and correct as translating it into some visual representation that would be incoherent to us, like a sequence of flashing lights or imperceptible noise patterns projected over an image of a dog.

I guess this is all a very long winded way of restating Chinese-room problem: we feed the man in the room a message; he returns one that, for all the world, is indistinguishable from a "real" response that you and I might send, but, like you said, he has no access to sense data. He also has no access to the biology underlying real mental processes. He also doesn't have any personhood that we can discern. He has, rather, gradually developed through reinforcement the tendency to provide responses approximating all of all of that.

I'm not sure the epistemological question "does he understand" (which is what the Chinese-room problem asks) has any meaningful answer. There's no mind, so there's no understanding. What there is, rather, is a system that generate patterns that we map to language and that our brains therefore map to communication, personhood, meaning, etc. It's the square I mentioned earlier. It's to us a convincing simulacrum, and it may be faithful enough to us to stand in those things, but that's not what it is.

My sense is that LLMs are (a) the big-data Pyramids of Giza and (b) a consequence of hardware and software developing ways of generating abstractions that capture and generate more complex patterns than were previously possible to capture or generate in a manner comprehensible to humans. Everything humans do follows some kind of pattern. Language is the perfect way for a machine to capture that, because is simpler than the world itself and has clear rules and patterns, encoded in representations computers already have, that, modeled well enough, can generate output indistinguishable from the real thing -- what you or I might do with it.

But the pattern matching that it does, and that we translate into language, is much more numerically rigorous and complex than anything you or I consciously do with words (hell, most people can't even figure out when to use "lie" vs "lay") and not doing what you or I do with it. It's not language. It's the square on the screen.