Comment by goatlover
1 day ago
We train them on the sum of human knowledge available in digital format, so I'm not sure that's an indictment, and the large models are incredibly complicated to the point it's hard to reverse engineer how they work so well and what they're actually doing to build internal models to predict the next token. Several billion to several trillion parameters is anything but simple.
Right, we needed compute on a large and fast scale to make LLMs actually work. But the basic principles are quite simple: train on a huge dataset for long enough, using some fairly simple ML maths but across a very large dimensional space, and then out the other side you get something that can competently use language, and is capable of doing useful work. This is somewhat surprising, no?
Yes, it is surprising.