Comment by antics
4 days ago
> “Google Translate is amazing, but it has taught us essentially nothing about linguistics,” Strogatz said. “We don’t understand why it works. It just works. That’s the Twilight Zone. It is knowledge without understanding.”
This is not true, and it is the crux of an enormous debate we're now having across many, many fields.
Classical Chomskyan syntax (which models sentences as parse trees) has been enormously, astonishingly predictive of the rich structure in real-world human language. But we are quick to forget that we invented concepts like "words" and "noun phrases" specifically to help us analyze and predict things about human language. What LLMs and modern MT shows, definitively, is that there is another, totally separate way to describe these phenomena, which is both vastly less descriptive, and also vastly more predictive.
That these appear to trade-off is unfortunate, but it is NOT the case that we "don't understand why it works." We absolutely understand why it works, if we didn't understand we would not be able to improve these systems. What we do not have is a clean, descriptive theory that boils this down to something that is formally specifiable and analyzable. But those are different things, and this is the root of the disconnect between many communities working in and around these spaces (linguists vs computational linguists; mathematicians vs computational mathematicians; mech interp people vs empiricists).
"We absolutely understand why it works, if we didn't understand we would not be able to improve these systems" - Understanding how to improve something does not require understanding how it works.
Growing a plant does not imply understanding soil minerals or photosynthesis. Training a dog does not imply understanding it's brain. Using LLMs to improve a codebase does not imply understanding C++. Improving a PC with new parts, does not imply understanding RAM.
You can understand abstractions about the system, without understanding the system. Without gradient descent or other forms of automatic optimisation, do you think humans could build something as capable as GPT-1?
Not sure if this will bake your noodle or not, but yes this is basically my point. Saying I understand how dog training works has very little overlap with saying I understand how the dog brain proteome works, even though they're related, and even though I might (or might not!) understand better how to train my dog for having fully understood the proteome.
Saying we understand training and deploying machine translation systems that have strong predictive (rather than descriptive) power, apparently has very little overlap with defining a formal, analyzable, mathematical model of language. Mathematician who refer to "understanding" generally mean the second thing, but the first thing is also a kind of understanding. Just messier.
I think they're referring to the fact that neural networks are more or less black boxes. Their internals are blurry and not explainable in the same way that traditional linguistics tries to explain things - with rules.
Of course, ultimately that's because language itself doesn't follow simple rules like linguists would prefer. They're more like guidelines.
Yes, that is one of the points I am making. Chomskyan syntax is a strongly descriptive theory that is entirely ex post. MT and LLM systems are strongly predictive and have no descriptive theory. It is completely wrong to say the first is "understanding" and the second is not. They are both different types of understanding. In particular, Chomskyan theory clearly does not represent enough "understanding" to predict real-world English sentences with similar lucidity to that of LLMs.
> LLM systems are strongly predictive
No, they're not predictive at all. They're generative. A prediction is a forecast based on a theoretical model about a state of affairs in the world. Making a guess what the weather tomorrow will be is a prediction, and a prediction can be true or false. An LLM generates text based on certain parameters, and output it generates aren't true or false, they're simply whatever the system is tuned to produce.
LLMs are terrible at making forecasts or world modelling precisely because they don't actually operate on much of any model or theory, they have no notion of truth or falsehood, they simply generate whatever you want them to generate.
2 replies →