Comment by NaiveEmpiricist
1 day ago
Indeed. Who'd have thought that mere statistics could do that? Hat's off to them - they made the darn thing work.
There's pretty interesting (I think) research in how effectively it's possible to scale down LLMs. If you're interested, take a look at the BabyLM conferences:
(I have no connection to them.) The work there demonstrates that gap between what we can do now with human-relevant amounts of training text, and what remains to be discovered. Quite a bit, it looks like.
"Mere statistics" is how physical reality works; I don't mean the language, the syntax
I mean shifting influence of gravity relative to an object with enough mass type ratios are everywhere
The language of math started as visual art to depict such ratios; in modern times this ground truth associations is frequently overlooked or never understood to exist in the first place without engagement
Differential diagnosis in medicine is exactly the same process; such processes and their successes are everywhere.
The issue we have in software is the outcome isn't as obvious as jabbing a blood draw needle in a patients eye; we spent a lot of time wasting time on illusory needs
“Mere statistics” is heavily underselling what the transformer architecture does. Unless you think “mere ion channels” is a sensible way to think about the human brain.