Comment by voidhorse
2 days ago
You should read some of the papers written in the 1940s and learn about the history of cybernetics. Your glowing perception of the "demon summoning" nature of ML might change a bit.
People want others to think this tech is mysterious. It's not. We've known the theory of these systems since the mid 1900s, we just didn't fully work out the resource arrangements to make them tractable until recently. Yes, there are some unknowns and the end product is a black box insofar as you cannot simply inspect source code, but this description of the situation is pure fantasy.
Good luck trying to use theory from the 1940s to predict modern ML. And if theory has little predictive power, then it's of little use.
There's a reason why so many "laws" of ML are empirical - curves fitted to experimental observation data. If we had a solid mathematical backing for ML, we'd be able to derive those laws from math. If we had solid theoretical backing for ML, we'd be able to calculate whether a training run would fail without actually running it.
People say this tech is mysterious because it is mysterious. It's a field where practical applications are running far ahead of theory. We build systems that work, and we don't know how or why.
We have solid backing in maths for it. But the fact is what we are seeking is not what the math told us, but an hope that what it told us is sufficiently close to the TRUTH. Hence the pervasive presence of errors and loss functions.
We know it’s not the correct answer, but better something close than nothing. (close can be awfully far, which is worse than nothing)
The math covers the low level decently well, but you run out of it quick. A lot of it fails to scale, and almost all of it fails to capture the high level behavior of modern AIs.
You can predict how some simple narrow edge case neural networks will converge, but this doesn't go all the way to frontier training runs, or even the kind of runs you can do at home on a single GPU. And that's one of the better covered areas.
2 replies →