Comment by zahlman
2 months ago
In an optical illusion, we perceive something that isn't there due to exploiting a correction mechanism that's meant to allow us to make better practical sense of visual information in the average case.
Asking LLMs to count letters in a word fails because the needed information isn't part of their sensory data in the first place (to the extent that a program's I/O can be described as "sense"). They reason about text in atomic word-like tokens, without perceiving individual letters. No matter how many times they're fed training data saying things like "there are two b's in blueberry", this doesn't register as a fact about the word "blueberry" in itself, but as a fact about how the word grammatically functions, or about how blueberries tend to be discussed. They don't model the concept of addition, or counting; they only model the concept of explaining those concepts.
I can't take credit for coming up with this, but LLMs have basically inverted the common Sci-Fi trope of the super intelligent robot that struggles to communicate with humans. It turns out we've created something that sounds credible and smart and mostly human well before we made something with actual artificial intelligence.
I don't know exactly what to make of that inversion, but it's definitely interesting. Maybe it's just evidence that fooling people into thinking you're smart is much easier than actually being smart, which certainly would fit with a lot of events involving actual humans.
Very interesting, cognitive atrophy is a serious concern that is simply being handwaved away. Assuming the apparent trend of diminishing returns continues, and LLMs retain the same abilities and limitations we see today, there's a considerable chance that they will eventually achieve the same poor reputation as smartphones and "iPad kids". "Chewing gum for the mind".
Children increasingly speak in a dialect I can only describe as "YouTube voice", it's horrifying to imagine a generation of humans adopting any of the stereotypical properties of LLM reasoning and argumentation. The most insidious part is how the big player models react when one comes within range of a topic it considers unworthy or unsafe for discussion. The thought of humans being in any way conditioned to become such brick walls is frightening.
The sci-fi trope is based on the idea of artificial intelligence as something like an electronic brain, or really just an artificial human.
LLMs on the other hand are a clever way of organising the text outputs of millions of humans. They represent a kind of distributed cyborg intelligence - the combination of the computational system and the millions of humans that have produced it. IMO it's essential to bear in mind this entire context in order to understand them and put them in perspective.
One way to think about it is that the LLM itself is really just an interface between the user and the collective intelligence and knowledge of those millions of humans, as mediated by the training process of the LLM.
Searle seems to have been right: https://en.m.wikipedia.org/wiki/Chinese_room
(Not that I am the first to notice this either)
From the wikipedia article:
> applying syntactic rules without any real understanding or thinking
It makes one wonder what comprises 'real understanding'. My own position is that we, too, are applying syntactic rules, but with an incomprehensibly vast set of inputs. While the AI takes in text, video, and sound, we take in inputs all the way down to the cellular level or beyond.
1 reply →
Celebrities, politicians and influencers are a constant reminder that people think others are far more intelligent than they actually are.
current gen AI is Pakleds of Star Trek TNG.
Give them a bit of power though, and they will kill you to take your power.
Moravec strikes again.
The real criticism should be the AI doesn't say "I don't know.", or even better, "I can't answer this directly because my tokenizer... But here's a python snippet that calculates this ...", so exhibiting both self-awareness of limitations combined with what an intelligent person would do absent that information.
We do seem to be an architectural/methodological breakthrough away from this kind of self-awareness.
For the AI to say this or to produce the correct answer would be easily achievable with post-training. That's what was done for the strawberry problem. But it's just telling the model what to reply/what tools to use in that exact situation. There's nothing about "self-awareness".
> But it's just telling the model what to reply/what tools to use in that exact situation.
So the exact same way we train human children to solve problems.
3 replies →