Comment by dismalaf

Because I can't see current techniques for creating LLMs fixing the pre-training problem. Right now big tech companies are training LLMs on, well, pretty much all human knowledge ever assembled, and they're still pretty dumb. They're wrong far too often and they don't have the capacity to learn and figure out things with a limited amount of data as humans do. Also, it's pretty clear that LLMs are flatlining.

Now, they are good text interfaces. They're good for parsing and creating text. There even seems to be very, very basic thought and maybe even creativity (at a very, very basic level). At this point though, I can't see them improving much more without a major change in technology, techniques, something. The first time I saw them I thought they were just regression analysis on steroids, and not going to lie, they still have that vibe considering tech companies have clusters up to 350k H100s and LLMs still are dumber than the average person for most tasks.

I'm currently creating an app that uses an LLM as an interface and it's definitely interesting, but most of the heavy lifting of the app will be the functions it calls and a knowledge database since it needs to have more concrete and current knowledge. But hey, it's nicer than implementing search from scratch I guess.