Comment by suddenlybananas
20 hours ago
The models require orders of magnitude more data to learn language than children do, and children's task is harder as they have to segment speech themselves.
20 hours ago
The models require orders of magnitude more data to learn language than children do, and children's task is harder as they have to segment speech themselves.
The comparison isn't so straightforward, since children's language-learning environment is multimodal and extremely rich. They're exposed not just to sequences of words, but also to behaviors, emotions, social interactions, and the physical world.
I think you just explained the situation, even if you didn't realize it.
LLMs process millions of times more text data and use millions of times more compute power to learn a language, mostly because we know how to do that technologically.
Children use much less text data, but also use a lot of other kinds of data, including interacting with other humans, to learn language. We have no idea how that works. And, on top of that, people can interact with pets of various species in the same way as children, but those pets never learn language.
So, something strange is going on that children can do, but that LLMs and puppies can't. Probably, part of it is that human children are much better at learning by copying what adults are doing, and part of it is that humans evolved to learn and use language. The details of what's going on there is what linguistics is about.
Yes, I believe arguments like GP vastly underestimate the amount of data involved in children's learning. All the data coming into their sensorium, from heard language to physical movement feedback, contributes to training all their skills, from physical movement to language.
A newborn brain is not a blank slate, it comes with a very rich prior (as Chomsky himself would argue), for example the ability to rapidly acquire language, control a body, etc.
An LLM must construct all of its grammatical structure, world models, etc. from scratch during pretraining.
LLM pre-training is more like recapitulating evolution of the mammalian brain than childhood development.
Of course children are not a blank state. My point is to argue that LLMs are not an argument against UG.
You're forgetting the millions of years of evolution that "pre-trained" child brains for this task.
I agree that language is innate.
Yes but it can be both innate and built on statistical models, contrary to Chomksy.
5 replies →
LLMs (at the autoregressive pretraining stage) are trained to predict what an (adult) human who could have said preceding words would have said next. You should add 15-30 years to the human training time.