← Back to context

Comment by GolfPopper

14 days ago

>Can LLMs actually parse human languages?

IMHO, no, they have nothing approaching understanding. It's Chinese Rooms[1] all the way down, just with lots of bell and whistles. Spicy autocomplete.

1. https://en.wikipedia.org/wiki/Chinese_room

32 comments

GolfPopper

Reply
EGreg  14 days ago

Actually, the LLMs made me realize John Searle’s “Chinese room” doesnt make much sense

Because languages have many similar concepts so the operator inside the Chinese room can understand nearly all the concepts without speaking Chinese.

And the LLM can translate to and from any language trivially, the inner layers do the actual understanding of concepts.

CamperBob2  14 days ago

Go ask the operator of a Chinese room to do some math they weren't taught in school, and see if the translation guide helps.

The analogy I've used before is a bright first-grader named Johnny. Johnny stumbles across a high school algebra book. Unless Johnny's last name is von Neumann, he isn't going to get anything out of that book. An LLM will.

So much for the Chinese Room.

  • jmb99  14 days ago

    > Go ask the operator of a Chinese room to do some math they weren't taught in school, and see if the translation guide helps.

    That analogy only holds if LLMs can solve novel problems that can be proven to not exist in any form in their training material.

    • CamperBob2  14 days ago

      They do. Spend some time using a modern reasoning model. There is a class of interesting problems, nestled between trivial ones whose answers can simply be regurgitated and difficult ones that either yield nonsense or involve tool use, that transformer networks can absolutely, incontrovertibly reason about.

      10 replies →

    • Workaccount2  14 days ago

      I think people give training data too much credit. Obviously it's important, but it also isn't a database of knowledge like it's made out to be.

      You can see this in riddles that are obviously in the training set, but older or lighter models still get them wrong. Or situations where the model gets them right, but uses a different method than the ones used in the training set.

  • dTal  14 days ago

    A "Chinese Room" absolutely will, because the original thought experiment proposed no performance limits on the setup - the Room is said to pass the Turing Test flawlessly.

    People keep using "Chinese Room" to mean something it isn't and it's getting annoying. It is nothing more than a (flawed) intuition pump and should not be used as an analogy for anything, let alone LLMs. "It's a Chinese Room" is nonsensical unless there is literally an ACTUAL HUMAN in the setup somewhere - its argument, invalid as it is, is meaningless in its absence.

    • CamperBob2  14 days ago

      A Chinese Room has no attention model. The operator can look up symbolic and syntactical equivalences in both directions, English to Chinese and Chinese back to English, but they can't associate Chinese words with each other or arrive at broader inferences from doing so. An LLM can.

      If I were to ask a Chinese room operator, "What would happen if gravity suddenly became half as strong while I'm drinking tea?," what would you expect as an answer?

      Another question: if I were to ask "What would be an example of something a Chinese room's operator could not handle, that an actual Chinese human could?", what would you expect in response?

      Claude gave me the first question in response to the second. That alone takes Chinese Rooms out of the realm of any discussion regarding LLMs, and vice versa. The thought experiment didn't prove anything when Searle came up with it, and it hasn't exactly aged well. Neither Searle nor Chomsky had any earthly idea that language was this powerful.

      2 replies →

  • codr7  14 days ago

    Give Johnny a copier and a pair of scissors and he will be able to perform more or less the same; and likely get more out of it as well, since he has a clue what he is doing.

  • geysersam  14 days ago

    How can you make that claim? Have you ever used an LLM that hasn't encountered high school algebra in it's training data? I don't think so.

    • Groxx  14 days ago

      I have at least encountered many LLMs with many school's worth of algebra knowledge, but fail miserably at algebra problems.

      Similarly, they've ingested human-centuries or more of spelling bee related text, but can't reliably count the number of Rs in strawberry. (yes, I understand tokenization is to blame for a large part of this. perhaps that kind of limitation applies to other things too?)

      2 replies →

  • xwolfi  14 days ago

    An LLM will get ... what exactly ? The ability to reorder its sentences ? The LLM doesn't think, doesn't understand, doesn't know what matters more than not, doesn't use what it learns, doesn't expand what it learns to new knowledge, doesn't enjoy reading that book and doesn't suffer through it.

    So what is it really gonna do with a book, that LLM ? Reorder its internal matrix to be a little bit more precise when autocompleting sentences sounding like the book ? We could build an nvidia cluster the size of the Sun and it would repeat sentences back to us in unbelievable ways but would still be unable to take a knowledge-based decision, I fear.

    So what are we in awe at exactly ? A pretty parrot.

    The day the Chinese room metaphor disappears is when ChatGPT replies to you that your question is so boring it doesn't want to expend the resources to think about it. But it'd be ready to talk about this or that, that it's currently trying to get better at. When it finally has agency over its own intelligence. When it acquires a purpose.

    • morsecodist  14 days ago

      This isn't really the meaning of the Chinese room. The Chinese room presupposes that the output is identical to that of a speaker who understands the language. It is not arguing that there is any sort of limit to what an AI can do with its output and it is compatible with the AI refusing to answer or wanting to talk about something else.