Comment by andai

Here's a short clip of Karpathy speaking on this subject.

https://youtu.be/UldqWmyUap4

Also this is the direction the small LLMs are moving in already. They are too small for general knowledge, but getting quite good at tool use (incl. Googling).

Now we just need them to be very strict about what they know and don't know! (I think this is still an open problem, even with big ones.)