Comment by TimTheTinker

This expresses an exact idea I've had for some time. Roughly, what's holding us back from more advanced AI (including AGI) is the complete lack of ontologies in modern AI systems (all the logical capabilities of old AI, like expert systems, logical solvers, and theorem provers).

Once an LLM-based system can be hard-locked to its own ontology, think and evaluate options and make decisions based on it, and update it based on what it learns from encounters with the real world, then I think we'll have made the final great breakthrough AI needs to become generally intelligent and useful.

In terms of Daniel Kahneman's Thinking, Fast and Slow, LLMs currently use only System 1 thinking (equivalent to fast, pattern-matched decisions humans make). You can scale that up to trillions of parameters, but fundamentally it's still fancy scaled-up probabilistic pattern matching. Like humans, fast decisions made by AI here only can be justified/audited in retrospect.

Integrated ontologies would move AI toward what Kahneman calls "System 2 thinking": slow, logical, systematic evaluation that is more auditable.