Comment by observationist
1 day ago
This is a serious misconception of human cognitive abilities.
We have the ability to abstract generally - there is no abstraction for which we lack the capacity to comprehend. We regularly visualize, contextualize, and satisfactorily explain systems with dozens of dimensions. The fact that we cannot hold 4,5+ spatial dimensions in our imaginations sufficiently to develop an intuition for navigation in that space and geometry does not logically extend to human brains lacking the wiring or hardware for systems of thinking that are beyond our capacity.
We do have limitations in scope, in both memory and speed. Both of these can be overcome with augmentation and interfacing with UI or direct neural connections, and intuitive, comprehensive, deep understanding of systems can be learned.
You could very well know the underlying theory of how your 8086 processor works, how it interfaces with all the elements of the motherboard, how electricity and physics interact at each level of abstraction from transistors to the pixels representing the spreadsheet you're using to do your taxes. You won't be able to simulate that in your head to any significant degree of resolution.
We will require similar levels of system thinking to acquire intuition and deep understanding of complex new theories and models. AI can assist with that by providing UI for useful levels of abstraction and segmenting theories into chunks we're capable of consuming. BCI and augmentation will definitely allow a more total, holistic understanding, and I think it's the augmentation path that will keep us competitive with AI.
There's also a huge issue with your use of the word subjective - math is objective. Proofs remain stable whether it's humans or any other system that does the processing. We test that objectivity by comparing the subjective readings from individual humans, and if the tests all return the same results, we can confidently say that the resulting proof is an objective fact about reality. Subjective fundamentally means that depending on the subject, the reading might change. Modern systems of math are formally, provably objective. That's how and why things are the way they are; if they weren't, people would experience radically different individuated realities, or there would be clusters of results shared across some measurable characteristic of the universe. That's not the case, so you can confidently say that the foundations of our math and logic are sound.
You can even prove it for yourself - the abductive chain of logic that allows you to contrast your own consciousness and subjective experience, determine that it comes about because your brain is wired to "do" consciousness, like all the other humans, and compare your subjective reporting of phenomenal experience with all the other reporting of phenomenal experience, and achieve a ridiculously high level of certainty, in the Bayes sense, that you and other humans are conscious; from that footing, you can confidently navigate the rest of enlightenment rationality and formal logic and mathematics.
At any rate, Egan's mistake is one of kind, but of scale - I am certain that as we formalize and start creating any sort of universal proof library, we will find that useful and interesting things are of necessity a tiny fraction of all possible valid formulations of any framework of logic and math. Crude attempts, such as OpenCyc and other formal ontological reasoner systems, would need trillions of low level rules to have a rough approximation of the world model as complex as that of a human child. AI with trillions of parameters could probably start getting to the point where there's parity with human scale, but even if you turned the entire planet earth into computronium and turned it toward the task of understanding all the theory and science of the universe, there will always be far more left to explore and understand than the sum total of all knowledge.
All that to say, humans will be fine with ergonomic interfaces that map to human capabilities, even for extraordinarily complex and hyperdimensional systems.
> There's also a huge issue with your use of the word subjective - math is objective. Proofs remain stable whether it's humans or any other system that does the processing. We test that objectivity by comparing the subjective readings from individual humans, and if the tests all return the same results, we can confidently say that the resulting proof is an objective fact about reality. Subjective fundamentally means that depending on the subject, the reading might change. Modern systems of math are formally, provably objective.
You are of course free to believe in mathematical Platonism, but that doesn't mean that non-Platonists would agree that proofs amount to objective "facts" about reality. And you are equally free to "prove it for yourself", which will just end up begging the question unless you are a Platonist.
That's not to say that math is subjective. But claiming that math is producing objective facts ignores at least a few hundred years of philosophy of mathematics (if not more). Even practicing mathematicians like Chaitin have described math as being more about inventing than discovering.
Holding a non-Platonist position also doesn't immediately lead to the sort of constructivist / "anything goes" position that some people ascribe to it, where you'd suddenly lose the ability to say that 2 + 2 = 4 and not 5. There are lots of philosophical positions that would agree that 2 + 2 is 4 without also claiming that this makes it a "fact" about any sort of objective reality or Platonic discovery.
> [...] claiming that math is producing objective facts ignores at least a few hundred years of philosophy of mathematics (if not more). Even practicing mathematicians like Chaitin have described math as being more about inventing than discovering.
Chaitin's work (including in his own opinion) moved the philosophical needle on maths towards "discovery" and away from "invention".
Indeed, his philosophy on the nature of mathematics is closer to "discovery", as in physics, than "invention" as in arbitrary game of rules and symbols (Hilbert) or language game (Wittgenstein).
I forget who said, "If you keep asking why, eventually you end up in the mathematics department." Calling it Platonism is a misunderstanding of how physics (and chemistry, biology, and the rest) emerge from mathematical structures and relationships that are more fundamental.
> there is no abstraction for which we lack the capacity to comprehend.
How could this ever be tested/falsified?
It feels a bit like "there is no idea we cannot think of." If we can't comprehend it, then it won't be an abstraction, it'll just be a mystery.
In principle - if you're able to scale appropriately, using technology to augment capacity, then in principle, there's no abstraction for which we lack the capacity to comprehend, because calculation is calculation. Turing Computers can calculate anything which can be calculated given enough time and memory. Brains are Turing complete.
It's not just a tautology, it's a feature of the universe- if it can be computed, it's comprehensible. Even quantum physics is just computation - truth tables and counterintuitive operators interacting over time in ways that are strange to our embodied norms, but nonetheless following rules and limits strictly defined by mathematics.
But again, that's in principle. It might be completely impractical - taking a million years for an individual human - to hold a particular idea in their head, while an advanced AI can have such thoughts many times a day. Such things would remain mysteries, but in principle, an augmented human, or a series of interfaces with the relevant abstraction levels of such an idea, theory, or system, would in principle give us comprehension.
In practice, we'll never run out of mystery or ignorance or mistakes.
Inconsistent logic.
You've made a grave assumption in treating this as a binary problem: comprehensible or incomprehensible. The truth is that this lies along a spectrum. I can "comprehend" a color which my eyes can't see but this is different than trying to comprehend red. Or as a variant, I can comprehend a color that I can't see naturally but if I go into a lab and they simulate those cones in very specific ways I won't ever be able to really imagine it. Seeing it will be clearly a different level of comprehension.
I fully agree with this but also for some reasons unmentioned. Truth has infinite precision, a thing we will never achieve. Similarly my namesake showed there are limits to axiomatic systems.
And there are problems that can't be solved by Turing complete machines. That only means they can solve computable problems but there's plenty that aren't. Very famously the halting problem isn't. And in the real world there's many problems which are intractable. And some are more intractable than others
It could potentially be falsified by an encounter with really weird aliens.
But I find it quite plausible because it feels like it's fundamentally "just" a restatement or a minor variation of the Church-Turing thesis.
I see it as implausible, and I find myself thinking of Godel's incompleteness theorem(s).
Somewhere out there are "correct" thoughts we can't think without changing how our brains work.
>> Crude attempts, such as OpenCyc and other formal ontological reasoner systems, would need trillions of low level rules to have a rough approximation of the world model as complex as that of a human child. AI with trillions of parameters could probably start getting to the point where there's parity with human scale, but even if you turned the entire planet earth into computronium and turned it toward the task of understanding all the theory and science of the universe, there will always be far more left to explore and understand than the sum total of all knowledge.
I thought CyC was a heroic attempt doomed to failure but not because it was rule-based. IIUC it used (custom) first-order logic and that's as expressive as expressive can be. There's no reason why a sufficiently large first-order rule-base could not capture every human concept. There's also no reason why "trillions of parameters" would be any better at that than "trillions of rules". It really comes down to what those rules or parameters are encoding.
What doomed CyC to failure, I think, is that its rule-base was mainly manually encoded. CyC was basically the world's biggest ever expert system, and it came with the biggest ever knowledge acquisition bottleneck. I don't think there's any magick to my claim, either. Human minds can't handle the complexity of a few dozen, let alone a few million, interconnected rules without making mistakes. It's hopeless trying to create such a system by hand. From a certain point onward you have no idea what your system can and can't do, because you have no idea what information it has and hasn't access to.
But it's hopeless trying to create such a system by chance, too, i.e. by feeding the system all the data we can find in the hope that it will somehow spontaneously acquire all the knowledge we want it to; much of which is not even in the deductive closure of the information we feed it (and LLMs are not deductive inference engines, unlike expert systems).
Some kind of automatic knowledge acquisition is clearly a much better idea than manually coding rules by hand, but I don't see how peta-scale machine learning has moved the needle much either. We're still stuck with systems that can do some spectacular things but can't do simple things. Or things that look simple to us.