Comment by foxyv

LLMs are essentially a text prediction engine. This can be used for basic reasoning tasks, however the LLM doesn't have much in the way of actual knowledge. However, it can use knowledge that exists to predict text better. For instance, if you plug a bunch of scientific articles into it, it will be really good at answering questions about the subject of those articles.

The problem is that context windows are very short. 200k-1M tokens or so. This means that the model needs to focus down on very specific information if possible. This is what makes tool using, reasoning, and agentic AI very powerful. The model can find the most relevant information it needs within its limited context and generate relevant answers to questions. The LLM pulls from web searches, documentation, long term memories in graph databases, and database queries to answer the questions using real information.