Comment by yosefk

Whichever way it's true in, it's not true in the sense that eg you can approximate any curve with a single layer neural net, and you're not actually going to be able to do it for problems CNNs or transformers work decently on. And Google indexed all of the public Internet way before its researchers came up with transformers.

Another way to look at it is that like you say, it was an old video but there has been progress since though we had large datasets when it came out by its own definition