Comment by lostmsu
1 day ago
Humans decidedly take much more data than LLMs. Like orders of magnitude. We happen to feed only distilled data to LLMs though. That's why they appear so efficient.
1 day ago
Humans decidedly take much more data than LLMs. Like orders of magnitude. We happen to feed only distilled data to LLMs though. That's why they appear so efficient.
If you are counting by bytes and including all sensory input, sure. In terms of amount of text, it's not close in the other direction. It's very difficult to make an apples to apples comparison here.
You can obviously count by bytes. They are bytes for the model or for a person. Bytes to bytes is apples to apples.
You can't count by text, because text is not the only thing children learn on. So unlike the bytes comparison the text comparison is totally invalid.
The BabyLM challenge has shown that multimodal models don't seem to learn in a more data-efficient way with respect to tokens. Adding an unlimited amount of images doesn't help you learn syntactic islands faster.
Which entry are you referring to?
Not a specific entry, but in four years, there's never been a clear advantage of multimodal models and the results have been extremely lackluckster, even when you allow unbounded images alongside a fixed quantity of text.
2 replies →