Comment by thomastjeffery
2 years ago
They get very near the point, and completely miss it at the end.
> It summarizes the standard arguments in the literature by a kind of super-autocomplete, refuses to take a stand on anything, pleads not merely ignorance but lack of intelligence
That is an accurate response from ChatGPT.
ChatGPT, indeed, does not offer intelligence. Calling it AI serves no purpose except personification.
The only thing ChatGPT does is "some kind of super autocomplete". That's all it can do.
The only reason ChatGPT is so impressive in the first place, is that the thing it is "autocompleting" - language itself - is impressive.
Language is data. Inside that data is not simply a collection of ideas: language holds the logical associations that connect ideas to each other. It can even hold an objective perspective on those associations. It can even hold an objective perspective on that objective perspective!
ChatGPT succeeds in finding new paths through that data, but it is walking blind: it doesn't know what any of the data is. It only knows where.
It just so happens that "where" holds a lot of meaning. Language uses "where" to encode a lot of information: "where" is a fundamental building block for grammar itself. It's called "semantics".
Without any other language feature, ChatGPT is able to manipulate and apply the "where" of language to itself. It's able to manipulate semantics.
But that's it. It can't do anything else. And that's a problem. What is semantically valid might be really useful, really wrong, or really nonsensical. The only filter ChatGPT has for those categories is what it has and hasn't read already. Most of that is not nonsense: most possible nonsense isn't written in the first place. Most of what we write has explicit meaning. But nothing stops us from writing silly things, or even lies. Nothing stops language itself from getting the subjects of the logic mixed up. As far as language is concerned, that is not a bug: it's a feature.
>it doesn't know what any of the data is. It only knows where.
And this right here is already driving a nail in the Chinese Room problem. At least from my interpretation of the problem that Searle presents, digital computers should not be able to do that at all, and yet here we are.
The situation isn't that mysterious or unknowable.
It's English that knows English. Chinese knows Chinese. The essence of grammar is encoded in the grammar itself: recursively.
Imagine the slabs of concrete that make up a sidewalk: between each of the slabs is a crack. Some slabs are shorter than others, so the distance between cracks isn't consistent.
Now imagine you took a string of pictures, each 1ft apart, all the way down the sidewalk, then stitched them together.
You show your friend the pictures. What do they see? A sidewalk.
ChatGPT gets a string of tokens: each token a few characters from the training dataset's text. That text is given in order. The boundaries between tokens are not in the same place as the boundaries between words, but they line up just as neatly.
Now imagine you shuffled the pictures, then stitched them back together. Does it still look like a sidewalk? Close enough. Some cracks are too close together or far apart to make sense, though.
With a handful of pictures, our sidewalk can go forever. And we can look at the original order to see what looks right and what doesn't.
If we avoid placing cracks closer together or farther apart than we saw them in the original, our sidewalk will look pretty good. If we try to repeat the original order, that's even better.
That's what ChatGPT does: it repeats what it knows in the order it has seen. The objects it is repeating are tokens, not words; but you can't tell that from the result.
But repeating text in "semantically familiar order" is how language is structured. Even if we didn't find or recognize words and subjects, we still get their effect, because the language already put that significance into the semantic order.
ChatGPT would be a meaningless continuation of nonsense if it wasn't trained on text that already contains language. But it was! Every token is a handful of meaning, neatly scooped out of a presorted list of semantic data. That order is preserved.
Even if the boundaries are drawn in the wrong place, the result looks just right, and we can see what we want to see.
>ChatGPT would be a meaningless continuation of nonsense if it wasn't trained on text that already contains language
I mean, if you put a child in a room and severely neglect them, you don't get a child that can speak any form of human language, and you'll find they an extremely underdeveloped brain.
1 reply →