Comment by krupan
1 day ago
"If a machine has to learn to understand humans to complete text, then that is what it has to do."
But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.
Your “and then” is doing a lot of work there. The steps between may or may not include some form of “learn to understand humans”, but you can’t just hide them behind “and then” if what we are doing is claiming some particular thing is not in the list.
Through training on human text, we are building implicitly in the weights a statistical model of what humans might write in response when presented with arbitrary pieces of text. It turns out that we can make these incredibly accurate.
If building an accurate internal model of something then using it to predict that thing’s behaviour is different to gaining understanding of that thing, we will need to pin down exactly what “understanding” means, or we are forever doomed to talk at cross purposes.
My "and then" simply implies order of operations. When it's fully "trained" then (and only then) can it generate text.
And I will reassert that even if it "understands" the text it was trained on, that is not the same as understanding humans. I mean really, we ARE humans and we barely understand humans.
The thing LLMs model and "predict" is simply, what words in what order are statistically common given these input words in this order.
You can write (non-ai) software to model and predict things using the laws of physics. I'd wager it would do a better job than any LLM at predicting where a rocket will go through space. Does that mean the program is conscious and "understands" physics? No
It can't even natively understand how many letters there are in words - how will it understand the meaning?
I wish people would do even the most basic amount of research into LLMs before opining about what they can or cannot do. There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning.
Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.
>The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence.
To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
4 replies →
> There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning. … Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'.
This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".
3 replies →
This is kind of a like assuming someone with bad spelling is stupid.
Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.
This is like saying because humans can't multiply 23472 by 1836736 in less than 5 nanoseconds that they can't possibly understand anything about maths.
You can't natively understand how many of your photoreceptors cells are activated by the period at the end of this sentence. How could you possibly understand the sentence's meaning?