Comment by stillpointlab
1 day ago
I find it hard to see how Chomsky has a leg to stand on.
> I understand how Chomsky might arrive at the conclusion that probabilistic models are unnecessary, from his study of the generation of language.
I think the point turns here for me. We have probabilistic models now that generate unbelievably convincing language. Norvig's following argument on the value of interpreting language statistically becomes secondary.
In order to stay moored in reality, our models must follow from evidence. I think it would take a peculiar obstinance to deny that the latest models show evidence of generating language.
The models require orders of magnitude more data to learn language than children do, and children's task is harder as they have to segment speech themselves.
I agree that is something we need to learn a lot more about, but a human child's brain does not appear to be necessary to learn language and I argue the evidence supports that.
The question isn't "are neural networks just like human brains?" It is "can statistical models generate and understand language?"
The answer to the actual question under discussion appears to be a resounding yes. It does not appear that learning language is innate in the way that Chomsky has argued.
It may be that learning language as efficiently as human children will involve some innate property that we have not yet discovered. But that is a perfect example of moving the goal posts. We go from "there is something special about the brain that allows us to learn language, something that cannot be reproduced statistically" to "there is something special about the brain that allows us to learn language efficiently, more than statistical models".
A newborn brain is not a blank slate, it comes with a very rich prior (as Chomsky himself would argue), for example the ability to rapidly acquire language, control a body, etc.
An LLM must construct all of its grammatical structure, world models, etc. from scratch during pretraining.
LLM pre-training is more like recapitulating evolution of the mammalian brain than childhood development.
Of course children are not a blank state. My point is to argue that LLMs are not an argument against UG.
The comparison isn't so straightforward, since children's language-learning environment is multimodal and extremely rich. They're exposed not just to sequences of words, but also to behaviors, emotions, social interactions, and the physical world.
I think you just explained the situation, even if you didn't realize it.
LLMs process millions of times more text data and use millions of times more compute power to learn a language, mostly because we know how to do that technologically.
Children use much less text data, but also use a lot of other kinds of data, including interacting with other humans, to learn language. We have no idea how that works. And, on top of that, people can interact with pets of various species in the same way as children, but those pets never learn language.
So, something strange is going on that children can do, but that LLMs and puppies can't. Probably, part of it is that human children are much better at learning by copying what adults are doing, and part of it is that humans evolved to learn and use language. The details of what's going on there is what linguistics is about.
Yes, I believe arguments like GP vastly underestimate the amount of data involved in children's learning. All the data coming into their sensorium, from heard language to physical movement feedback, contributes to training all their skills, from physical movement to language.
You're forgetting the millions of years of evolution that "pre-trained" child brains for this task.
I agree that language is innate.
7 replies →
LLMs (at the autoregressive pretraining stage) are trained to predict what an (adult) human who could have said preceding words would have said next. You should add 15-30 years to the human training time.
The hardest theory to disprove is one an entire field livelihood rests upon