Comment by YeGoblynQueenne
2 years ago
>> Think of all the data that has entered all your senses in your entire lifetime. More than goes into ChatGPT, I'll tell you that.
The question is how much of that was only text data, or only language anyway. Th e answer is- not that much, really. Chomsky's famous point about "the poverty of the stimulus" was based on research that showed human children learn to speak their native languages from very few examples of it spoken by the adults around them. They certainly don't learn from many petabytes of text as in the entire web.
If you think about it, if humans relied on millions of examples to learn to speak a language we would never have learned to speak in the first place. Like, back whenever we started speaking as a species. There was certainly nothing like human language back then, so there weren't any examples to learn from. Try that for "zero-shot learning".
Then again, there's the issue that there are many, many animals that receive the same, or even richer, "data" from their senses throughout their lives, and still never learn to speak a single word.
Humans don't just learn from examples, and the way we learn is nothing like the way in which statistical machine learning algorithms learn from examples.
Thinking about it as "text data" is both your and Chomsky's problem -- the >petabytes of data aren't preprocessed into text. They're streams of sensory input. It's not zero shot if it's years of data of observing human behavior through all your senses.
Other animals receiving data and not speaking isn't a good line of argument, I think. They could have very different hardware or software in their brains, and have completely different life experiences and therefore receive very different data. Notably, where animals and humans do have much potentially learned (or learned through evolution) behavior in common -- such as pathfinding, object detection, hearing, and high level behaviors like seeking food and whatever else.
>> Thinking about it as "text data" is both your and Chomsky's problem -- the >petabytes of data aren't preprocessed into text. They're streams of sensory input. It's not zero shot if it's years of data of observing human behavior through all your senses.
I'm a little unsure what you mean. I think you mean that humans learn language not just from examples of language, but from examples of all kinds of concepts in our sensory input, not just language?
Well, that may or may not be the case for humans, but it's certainly not the case for machine learning systems. Machine learning systems must be trained with examples of a particular concept, in order to learn that concept and not another. For instance, language models must be trained with examples of language, otherwise they can't learn language.
There are multi-modal systems that are trained on multiple "modalities" but they can still not learn concepts for which they are not given specific examples. For instance, if a system is trained on examples of images, text and time series, it will learn a model of images, text and time series, but it won't be able to recognise, say, speech.
As to whether humans learn that way: who says we do? Is that just a conjecture proposed to support your other points, or is it something you really think is the case, and believe, based on some observations etc?
I think you’re missing the meat of my point. The stuff LLMs are trained on is in no way similar to what human brains have received. It’s a shortcut to train them directly on text tokens. Because that’s the data we have easily available. But it doesn’t mean the principles of machine learning (which are loosely derived from how the brain actually works) apply only to text data or narrow categories of data like you mentioned. It just might require significantly more and different input data and compute power to achieve more generally intelligent results.
What I believe personally is I don’t think there is any reason to rule out that the basics of neural networks could serve as the foundation of artificial general intelligence. I think a lot of the criticism of this sort of technology being too crude to do so is missing the forest for the trees.
I have a brain and it learns and I’ve watched many other people learn too and I see nothing there that seems fundamentally distinct from how machine learning behaves in very general terms. It’s perfectly plausible that my brain has just trained itself on all the sensory data of my entire life and is using that to probabilistically decide the next impulse to send to my body in the same way an LLM predicts the most appropriate next word.
1 reply →
Not OP, but I'm not convinced by the talking point that a baby has an equivalent or greater petabytes of data because they are immersed in a sensory world. I can't quite put my finger on it but my feeling is that that line of reasoning contains a kind of category error. Maybe I'll wake up tomorrow and have a clearer idea of my objection, but I've seen your talking point echoed by many others as well, and this interests me.
What is all the “video” and “audio” and other sensory input but petabytes of data streaming into your brain? Seems like a pretty objectively measurable concept, right?