Comment by vannevar
1 year ago
I would argue that Lenat was at least directionally correct in understanding that sheer volume of data (in Cyc's case, rules and facts) was the key in eventually achieving useful intelligence. I have to confess that I once criticized the Cyc project for creating an ever-larger pile of sh*t and expecting a pony to emerge, but that's sort of what has happened with LLMs.
I suspect at some point the pendulum will again swing back the other way and symbolic approaches will have some kind of breakthrough and become trendy again. And, I bet it will likely have something to do with accelerating these systems with hardware, much like GPUs have done for neural networks, in order to crunch really large quantities of facts
The Bitter Lesson has a few things to say about this.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
The Bitter Lesson says "general methods that leverage computation are ultimately the most effective". That doesn't seem to rule out symbolic approaches. It does rule out anything which relies on having humans in the loop, because terabytes of data plus a dumb learning process works better than megabytes of data plus expert instruction.
(I know your message wasn't claiming that The Bitter Lesson was explicitly a counterpoint, I just thought it was interesting.)
Imho, this is wrong. Even independent of access to vast amounts of compute, symbolic methods seem to consistently underperform statistical/numerical ones across a wide variety of domains. I can't help but think that there's more to it than just brute force.
1 reply →
Real AGI will need a way to reason about factual knowledge. An ontology is a useful framework for establishing facts without inferring them from messy human language.
These guys are trying to combine symbolic reasoning with LLMs somehow: https://www.symbolica.ai/
check out Imandra's platform for neurosymbolic AI - https://www.imandra.ai/
Or maybe program synthesis combined by LLMs might be the way?
It does seem like the Cyc people hit the wall with simply collecting facts. Having to have a human in the loop.
The problem I think is if you have LLMs figuring out the propositions, the whole system is just as prone to garbage-in-garbage-out as LLMs are.
But
a) The pile of LLM training data is vastly larger. b) The data is actual human utterances in situ--these are ponies, not pony shit. c) LLMs have no intelligence ... they channel the intelligence of a vast number of humans by pattern matching their utterances to a query. This has indeed proved useful because of how extremely well the statistical apparatus works, but the fact that LLMs have no cognitive states puts great limits on what this technology can achieve.
With Cyc, OTOH, it's not even clear what you can get out of it. The thing may well be useful if combined with LLMs, but it's under lock and key.
The big conclusions about symbolic AI that the author reaches based on this one system and approach are unwarranted. As he himself notes, "Even Ernest Davis and Gary Marcus, highly sympathetic to the symbolic approach to AI, found little evidence for the success of Cyc, not because Cyc had provably failed, but simply because there was too little evidence in any direction, success or failure."
>> they channel the intelligence of a vast number of humans by pattern matching their utterances to a query.
Just a little problem with that: to understand the utterances of a vast number of humans you need to channel it to something that can understand the utterances of humans in the first place. Just channeling it around from statistic to statistic doesn't do the trick.
Um, the "something" is the person reading the LLM's output. I'm afraid that you have completely missed the context and point of the discussion, which was not about LLMs understanding things--they understand nothing ("LLMs have no cognitive states"). But again, "because of how extremely well the statistical apparatus works", their outputs are useful to intelligent consumers who do have cognitive states--us.
The dataset for lots of LLMs is literally called "The Pile" lmao
That’s hilarious, but at least Llama was trained on libgen, an archive of most books and publications by humanity, no? Except for the ones which were not digitized I guess
So there is probably a big pile of Reddit comments, twitter messages, and libgen and arxiv PDFs I imagine
So there is some shit, but also painstakingly encoded knowledge (ie writing), and yeah it is miraculous that LLMs are right as often as they are
libgen is far from an archive of "most" books and publications, not even close.
The most recent numbers from libgen itself are 2.4 million non-fiction books and 80 million science journal articles. The Atlantic's database published in 2025 has 7.5 million books.[0] The publishing industry estimates that many books are published each year. As of 2010, Google counted over 129 million books[1]. At best an LLM like Llama will have have 20% of all books in its training set.
0. https://www.theatlantic.com/technology/archive/2025/03/libge...
1. https://booksearch.blogspot.com/2010/08/books-of-world-stand...
On libgen.mx they claim to have 33,569,200 books and 84,844,242 articles
1 reply →
It's a miracle, but it's all thanks to the post-training. When you think of it, for so-called "next token predictors", LLMs talk in a way that almost no one actually talks, with perfect spelling and use of punctuation. The post-training somehow is able to get them to predict something along the lines of what a reasonably intelligent assistant with perfect grammar would say. LLMs are probably smarter than is exposed through their chat interface, since it's unlikely the post-training process is able to get them to impersonate the smartest character they'd be capable of impersonating.
I dunno I actually think say Claude AI SOUNDS smarter than it is, right now
It has a phenomenal recall. I just asked it about "SmartOS", something I knew about, vaguely, in ~2012, and it gave me a pretty darn good answer. On that particular subject, I think it probably gave a better answer than anyone I could e-mail, call, or text right now
It was significantly more informative than wikipedia - https://en.wikipedia.org/wiki/SmartOS
But I still find it easy to stump it and get it to hallucinate, which makes it seem dumb
It is like a person with good manners, and a lot of memory, and which is quite good at comparisons (although you have to verify, which is usually fine)
But I would not say it is "smart" at coming up with new ideas or anything
I do think a key point is that a "text calculator" is doing a lot of work ... i.e. summarization and comparison are extremely useful things. They can accelerate thinking
https://ai-2027.com/ postulates that a good enough LLM will rewrite itself using rules and facts... sci-fi, but so is chatting with a matrix multiplication.
I doubt it. The human mind is a probabilistic computer, at every level. There’s no set definition for what a chair is. It’s fuzzy. Some things are obviously in the category, and some are at the periphery of it. (Eg is a stool a chair? Is a log next to a campfire a chair? How about a tree stump in the woods? Etc). This kind of fuzzy reasoning is the rule, not the exception when it comes to human intuition.
There’s no way to use “rules and facts” to express concepts like “chair” or “grass”, or “face” or “justice” or really anything. Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
> There’s no set definition for what a chair is.
Sure there is: a chair is anything upon which I can comfortably sit without breaking it.
18 replies →
So you're just ignoring all the probabilistic, fuzzy etc. Prologs etc. which do precisely that? https://github.com/lab-v2/pyreason
> Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
The counterposition to this is no more convincing: cognition is fuzzy, but it's not really clear at all that it's probabilistic: I don't look at a stump and ascertain its chairness with a confidence of 85%, for example. The actual meta-cognition of "can I sit on this thing" is more like "it looks sittable, and I can try to sit on it, but if it feels unstable then I shouldn't sit on it." In other words, a defeasible inference.
(There's an entire branch of symbolic logic that models fuzziness without probability: non-monotonic logic[1]. I don't think these get us to AGI either.)
[1]: https://en.wikipedia.org/wiki/Non-monotonic_logic
13 replies →
> The human mind is a probabilistic computer, at every level.
Fair enough, but an airplane's wing is not very similar to a bird's wing.
3 replies →
> There’s no way to use “rules and facts” to express concepts like “chair” or “grass”, or “face” or “justice” or really anything. Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
Are you sure? In terms of theoretical foundations for AGI, AIXI is probabilistic but godel-machines are proof based and I think they'd meet criteria for deterministic / symbolic. Non-monotonic and temporal logics also exist, where chairness exists as a concept that might be revoked if 2 or more legs are missing. If you really want to get technical then by allowing logics with continuous time and changing discrete truth values, then you can probably manufacture a fuzzy logic where time isn't considered but truth/certainty values are continuous. Your ideas about logic might be too simple, it's more than just Aristotle
2 replies →
> This kind of fuzzy reasoning is the rule, not the exception when it comes to human intuition.
That is indeed true. But we do have classic fuzzy logic, and it can be used to answer these questions. E.g. a "stool" maybe a "chair", but "automobile" is definitely not.
Maybe the symbolic logic approach could work if it's connected with ML? Maybe we can use a neural network to plot a path in the sea of assertions? Cyc really seems like something that can benefit the world if it's made open under some reasonable conditions.
1 reply →
"The human mind is a probabilistic computer, at every level."
We don't know that. It's mostly probabilistic. That innate behavior exists suggests some parts might be deterministic.
Words are used due to the absence of things. They fill an immediate experiential void and stand in for something else, because you want or need another person to evoke some fantasy to fill this absence and make understanding possible.
If you have a mind and it is a computer, then it is because of nurture, because the brain is nothing like a computer, and computers simulating language are nothing like brains.
That is not what is suggested. Llm still fuzzy mess, but supervisor / self editing is rules based
The way I see it:
(1) There is kind of a definition of a chair. But it's very long. Like, extremely long, and includes maybe even millions to billions of logical expressions, assuming your definition might need to use visual or geometric features of a given object to be classified as a chair (or not chair).
This is a kind of unification of neural networks (in particular LLMs) and symbolic thought: large enough symbolic thought can simulate NNs and vice versa. Indeed even the fact that NNs are soft and fuzzy does not matter theoretically, it's easy to show logical circuits can simulate soft and fuzzy boundaries (in fact, that's how NNs are implemented in real hardware! as binary logic circuits). But I think specific problems have varying degrees of more natural formulation as arithmetic, probabilistic, linear or fuzzy logic, on one hand, and binary, boolean-like logic on the other. Or natural formulations could involve arbitrary mixes of them.
(2) As humans, the actual definitions (although they may be said to exist in a certain way at a given time[1]) vary with time. We can, and do, invent new stuff all the time, and often extend or reuse old concepts. For example, I believe the word 'plug' in english likely well predates modern age, probably used to refer to original electrical power connectors. Nowadays there are USB plugs, which may not carry power at all, or audio plugs, etc. (maybe there are better examples). In any case the pioneer(s) usually did not envision all a name could be used for, and uses evolve.
(3) Words are used as tools to allow communication and, crucially, thought. There comes a need to put a fence (or maybe a mark) in abstract conceptual and logic space, and we associate that with a word. Really a word could be "anything we want to communicate", represent anything. In particular changes to the states of our minds, and states themselves. That's usually too general, most words are probably nouns which represent classifications of objects that exist in the world (like the mentioned chair) -- the 'mind state' definition is probably general enough to cover words like 'sadness', 'amazement', etc., and 'mind state transitions' probably can account for everything else.
We use words (and associated concepts) to dramatically reduce the complexity of the world to enable or improve planning. We can then simplify our tasks into a vastly simpler logical plan: even something simple like put shoes, open door, go outside, take train, get to work -- without segmenting the world into things and concepts (it's hard to even imagine thought without using concepts at all -- it probably happens instinctively), the number of possibilities involved in planning and acting would be overwhelming.
Obligatory article about this: https://slatestarcodex.com/2014/11/21/the-categories-were-ma...
---
Now this puts into perspective the work of formalizing things, in particular concepts. If you're formalizing concepts to create a system like Cyc, and expect it to be cheap, simple, reliable, and function well into the future, by our observations that should fail. However, formalization is still possible, even if expensive, complex, and possibly ever changing.
There are still reasons you may want to formalize things, in particular to acquire a deeper understanding of those things, or when you're okay in creating definitions set in stone because they will be confined to a group being attentive and restrictive to their formal definitions (and not, as natural language, evolving organically according to convenience): that's the case with mathematics. The peano axioms still define the same natural numbers; and although names may be reused, you can usually specify them to a particular axiomatic definition that will never change. And thus we can keep building facts on those foundations forever -- while what is a 'plug' in natural language might change (and associated facts about plugs become invalid), we can define mathematical objects (like 'natural numbers') with unchanging properties, and ever-valid and potentially ever-growing valid facts to be known about them, reliably. So fixing concepts in stone more or less (at least when it comes to a particular axiomatization) is not such a foolish endeavor it may look like, quite the opposite! Science in general benefits from those solid foundations.
I think eventually even some concepts related to human emotions and specially ethics will be (with varying degrees of rigor) formalized to be better understood. Which doesn't mean human language should (or will) stop evolving and being fuzzy, it can do so independently of formal more rigid counterparts. Both aspects are useful.
[1] In the sense that, at a given time, you could (theoretically) spend an enormous effort to arrive at a giant rule system that would probably satisfy most people, and most objects referred to as chairs, at a given fixed time.
[dead]
How will the rules and facts be connected? By some discrete relationship? This stuff only works for math, and is the basis for the bitter lesson.
Intelligence is compression, and this is the opposite of that