Comment by accurrent
10 hours ago
It's still important to support research in this direction cause us meatbags cost a LOT less energy to train than GPTs even if you assume that it takes 30 years to train a PhD. Our current training methods honestly leave a lot to be desired. We almost certainly dont do full backpropagation.
I’m not sure that’s true. We have literal billions of years of training.
Human "pre-training" is vastly underappreciated.
Hell, animals are an even better example. Many animals pop out of the womb and start walking and eating and acting just like an adult!
One could argue that humans are more adaptable to novel environments
If you wanna get philosophical then the decades of research that humans did is also AI "pre-training". And all of that is built on knowledge that humans have spent our whole existence amassing so maybe all of human "pre-training" is also AI pre-training.
A fun but silly exercise.
I'm sure we don't literally do backpropagation, but differential equations that settle toward stationary states show up a lot in biology
You need to factor in the cost of training all those PhDs that never end up producing much of interest though you don't get to pick the best afterwards and claim all it took was to train him/her. Plus, once trained, it's productive only for 6 subjective hours per day (including weekends and holidays). Might have costed gazillions to train a frontier LLM but it works for millions of hours/day.
PHD inference doesn't scale, though.
I think it's fun back-of-the-napkin sometimes to compare meat to matmuls but ultimately fallacious to its core, making it an intellectual tarpit.
It's much harder to argue with the math and empirical results.
Energy cost may be a pointless comparison since we don't eat electricity, but it's very reasonable to look at sample efficiency and ask what we have there that our theories miss.