Most of Wolfram's discussion of "bugs" is not about the following situation:
You want a program that does X. So you think about what a program that does X should look like, and write a program that does what you think should produce result X, but maybe you made some mistakes.
but about the following, to my mind completely different, situation:
You want a program that does X. So you write a bunch of random short programs and after a while you find one that looks on small examples as if it's doing X. You use that one, but maybe actually it doesn't always work.
The key difference here is that in the first case you try to make there not be bugs by understanding what the program is doing and in the second you try to make there not be bugs by looking at the program's output.
(Perhaps some super-extreme practitioners of test-driven development might argue for something like the latter. Perhaps some AI-driven programming is uncomfortably close to it. But it's not, generally, how people have produced and debugged software.)
Wolfram kinda tries to justify this perspective by saying:
"Fundamentally the only way to make sure a program will always do what you want is to understand everything it can do. But there’s a kind of paradox implicit in that: if you can really understand everything your program can do that basically means the program is doing something computationally reducible, and you probably in the end didn’t actually need to run the program with all its steps to get the result you wanted."
but I think this is silly (and suspect it's the result of Wolfram trying to shoehorn his pet notion of "computational (ir)reducibility" into places where it brings no actual insight). It happens all the time that you write a program that (at least in principle) you understand completely, but that you don't know a much cheaper way to do it.
Computer scientists generally struggle with intensional contexts, ie., the sense in which sqrt(4) isnt (1 + 1); or the sense in which I believe(that sqrt(1101) ~10) but I dont believe(that 10^2 ~1101)
So they replace everything with extensional contexts, ie., in/out sets -- where things are replaceable by their values. And then use the machinery of very simple mathematics to analyse the situation.
In ML, consider comparing two classifiers extensionally, so that two classifiers "are the same" if they partition some input data exactly the same vs., declaring them different if they do so by means of some differing internal representation.
Since intensional contexts are the archetypal feature of mental representations, they're rarely analysed well by computer scientists.
At the risk of straying into forbidden metacommentary: look, I understand (and also feel) the temptation to have a go at Stephen Wolfram every time he posts something wacky, but… he’s just a guy with a fun hobby, isn’t he? Sure it’s all a bit eccentric and self-indulgent, but so are most hobbies, and in contrast to basically every other rich tech man he isn’t actually hurting anyone. Live and let live I reckon.
One of the frustrating things with the way Stephen Wolfram works is because he never actually defines anything it’s very hard to pin down what is actually interesting empirical science and what is data visualisation buggering around.
Here, before he starts, how do we know that the bugs in the Turing machine implementation of f(n)=n+1 are in any way representative of bugs in a normal computation? He seems to be generating random state machines and then evaluating them and picking ones which are nearly but not quite correct implementations of f(n)=n+1. Is that like a normal “honest” software bug? Is it like a deliberate software sabotage like you might look for when evaluating a software supply chain risk? Is it like the sort of vulnerability you might see when fuzzing? As far as I can see, there’s no reason to think it’s like any of these.
More broadly, since the “Principle of Computational Equivalence” and “computational irreducibility” are referred to as touchstones but are never actually defined or properly established how are we supposed to treat them? Interesting conjectures? Actual axioms? Something in between?
I see it as related to the Church-Turing thesis which is also a bit fuzzy but I think comes down to the conjecture that there is no physical computer which exceeds the Turing machine in capabilities. (Quantum computers, as we know them, don't count because they can be simulated by classical computers with sufficient space and time)
One thing I find tragic about Wolfram is that he found his holy grail, Rule 30, at the very beginning of his project and ever since he's found that every family of rules that he's look at that is sufficiently powerful has something like Rule 30 -- but always a bit more complex, complicated and not quite so beautiful, in fact often outright ugly...
... so he jumps into this sort of thing and samples of some of a ruliad, finds the usual things he finds in every ruliad, gets excited about it and thinks he has a "theory" and then moves on to something else and most of us are scratching our heads.
The Church-Turing hypothesis I agree could be crisper (ie the definition of effectively computable is a bit complicated/weasely), but it actually says something quite concrete that I can understand and potentially falsify ie that a universal turing machine, the lambda calculus and general recursion can all "effectively compute" the same things. That seems reasonable, and I can check that something I can write in lambda calculus I can compute on a Turing Machine (simulated at least).
Whereas in stating the Principle of Computational Equivalence Wolfram says things like "... the evolution of weather systems can, in principle, compute the same things as a computer."[1]
Well, ... wtf? What does it mean to say the evolution of a weather system can compute something? Where do we even start on understanding this? I pick up a calculator and compute the volume of a cone. It's a sunny day outside. How do I "in principle" use that to check my calculation? It literally makes no sense whatsoever on its face as a claim but it is so slippery because of that "in principle" that it doesn't give me anything to hold onto that I could even test.
And that's before I even get started on the "underpants gnome" logical justification behind the claim, which seems to be:
1. Computers are complex
2. Weather is complex
3. Therefore they are "in some sense" equivalent
4. <don't worry about step 4>
5. Thus weather can "in principle" compute the same things as a computer.
I'm really not trying to misrepresent him, I'm just baffled.
Am I wrong to conclude that the statement by Google's VP of Security Engineering that "we simply must eliminate every software vulnerability on Earth" (before AI Agents find them) is not only stunningly ambitious, but doomed from the start?
'Ruliology' is an ignorant neologism, and whoever thought it up should be dropped down a well. Canonology is linguistically consistent without being obscure.
I went through a process of “what the fuck is this bullshit” to “ok, yes, that makes sense and can suffice for my multiverse worldview until someone writes a better one” when I read all his foundational essays on the topic.
In this framing, clean coding abstractions gain a purpose beyond helping humans understand the code and actually help enforce that the end result is reasonably bug free. Building code out of understandable pieces can ensure that the whole thing is reasonably understandable.
While the article is focused on “low level” bugs, the key insight is that human understanding has limits, and bugs show up when the human is overconfident in their understanding, and unwilling to verify the code by running it on a sample of test cases. This does seem to help explain why “high level” bugs occur, where the reality somehow doesn’t match the assumption. Maybe a dependency is on the wrong version, a global variable isn’t initialized in certain cases, etc.
In a world of LLMs producing code that isn’t always understandable to humans, it does raise the question of how to enforce that AI code is bug free, and maybe forcing coding agents to stick to human abstractions is the missing piece.
Most of Wolfram's discussion of "bugs" is not about the following situation:
You want a program that does X. So you think about what a program that does X should look like, and write a program that does what you think should produce result X, but maybe you made some mistakes.
but about the following, to my mind completely different, situation:
You want a program that does X. So you write a bunch of random short programs and after a while you find one that looks on small examples as if it's doing X. You use that one, but maybe actually it doesn't always work.
The key difference here is that in the first case you try to make there not be bugs by understanding what the program is doing and in the second you try to make there not be bugs by looking at the program's output.
(Perhaps some super-extreme practitioners of test-driven development might argue for something like the latter. Perhaps some AI-driven programming is uncomfortably close to it. But it's not, generally, how people have produced and debugged software.)
Wolfram kinda tries to justify this perspective by saying:
"Fundamentally the only way to make sure a program will always do what you want is to understand everything it can do. But there’s a kind of paradox implicit in that: if you can really understand everything your program can do that basically means the program is doing something computationally reducible, and you probably in the end didn’t actually need to run the program with all its steps to get the result you wanted."
but I think this is silly (and suspect it's the result of Wolfram trying to shoehorn his pet notion of "computational (ir)reducibility" into places where it brings no actual insight). It happens all the time that you write a program that (at least in principle) you understand completely, but that you don't know a much cheaper way to do it.
Computer scientists generally struggle with intensional contexts, ie., the sense in which sqrt(4) isnt (1 + 1); or the sense in which I believe(that sqrt(1101) ~10) but I dont believe(that 10^2 ~1101)
So they replace everything with extensional contexts, ie., in/out sets -- where things are replaceable by their values. And then use the machinery of very simple mathematics to analyse the situation.
In ML, consider comparing two classifiers extensionally, so that two classifiers "are the same" if they partition some input data exactly the same vs., declaring them different if they do so by means of some differing internal representation.
Since intensional contexts are the archetypal feature of mental representations, they're rarely analysed well by computer scientists.
At the risk of straying into forbidden metacommentary: look, I understand (and also feel) the temptation to have a go at Stephen Wolfram every time he posts something wacky, but… he’s just a guy with a fun hobby, isn’t he? Sure it’s all a bit eccentric and self-indulgent, but so are most hobbies, and in contrast to basically every other rich tech man he isn’t actually hurting anyone. Live and let live I reckon.
One of the frustrating things with the way Stephen Wolfram works is because he never actually defines anything it’s very hard to pin down what is actually interesting empirical science and what is data visualisation buggering around.
Here, before he starts, how do we know that the bugs in the Turing machine implementation of f(n)=n+1 are in any way representative of bugs in a normal computation? He seems to be generating random state machines and then evaluating them and picking ones which are nearly but not quite correct implementations of f(n)=n+1. Is that like a normal “honest” software bug? Is it like a deliberate software sabotage like you might look for when evaluating a software supply chain risk? Is it like the sort of vulnerability you might see when fuzzing? As far as I can see, there’s no reason to think it’s like any of these.
More broadly, since the “Principle of Computational Equivalence” and “computational irreducibility” are referred to as touchstones but are never actually defined or properly established how are we supposed to treat them? Interesting conjectures? Actual axioms? Something in between?
I see it as related to the Church-Turing thesis which is also a bit fuzzy but I think comes down to the conjecture that there is no physical computer which exceeds the Turing machine in capabilities. (Quantum computers, as we know them, don't count because they can be simulated by classical computers with sufficient space and time)
One thing I find tragic about Wolfram is that he found his holy grail, Rule 30, at the very beginning of his project and ever since he's found that every family of rules that he's look at that is sufficiently powerful has something like Rule 30 -- but always a bit more complex, complicated and not quite so beautiful, in fact often outright ugly...
... so he jumps into this sort of thing and samples of some of a ruliad, finds the usual things he finds in every ruliad, gets excited about it and thinks he has a "theory" and then moves on to something else and most of us are scratching our heads.
The Church-Turing hypothesis I agree could be crisper (ie the definition of effectively computable is a bit complicated/weasely), but it actually says something quite concrete that I can understand and potentially falsify ie that a universal turing machine, the lambda calculus and general recursion can all "effectively compute" the same things. That seems reasonable, and I can check that something I can write in lambda calculus I can compute on a Turing Machine (simulated at least).
Whereas in stating the Principle of Computational Equivalence Wolfram says things like "... the evolution of weather systems can, in principle, compute the same things as a computer."[1]
Well, ... wtf? What does it mean to say the evolution of a weather system can compute something? Where do we even start on understanding this? I pick up a calculator and compute the volume of a cone. It's a sunny day outside. How do I "in principle" use that to check my calculation? It literally makes no sense whatsoever on its face as a claim but it is so slippery because of that "in principle" that it doesn't give me anything to hold onto that I could even test.
And that's before I even get started on the "underpants gnome" logical justification behind the claim, which seems to be:
1. Computers are complex
2. Weather is complex
3. Therefore they are "in some sense" equivalent
4. <don't worry about step 4>
5. Thus weather can "in principle" compute the same things as a computer.
I'm really not trying to misrepresent him, I'm just baffled.
[1] https://mathworld.wolfram.com/PrincipleofComputationalEquiva...
It's because he's an intellectual con artist.
Am I wrong to conclude that the statement by Google's VP of Security Engineering that "we simply must eliminate every software vulnerability on Earth" (before AI Agents find them) is not only stunningly ambitious, but doomed from the start?
https://youtu.be/B_7RpP90rUk (at 3:00)
Shhhhh!
It may be the only way to keep us all employed.
Kurt Godel says that you are correct.
Eh? I don’t see how Gödel’s results imply that.
'Ruliology' is an ignorant neologism, and whoever thought it up should be dropped down a well. Canonology is linguistically consistent without being obscure.
It's Wolfram-speak, so you'd want to drop Stephen if you're committed.
It's part of his GUT. You will likely become committed to the well when you encounter ruliad: https://writings.stephenwolfram.com/2021/11/the-concept-of-t...
I went through a process of “what the fuck is this bullshit” to “ok, yes, that makes sense and can suffice for my multiverse worldview until someone writes a better one” when I read all his foundational essays on the topic.
In this framing, clean coding abstractions gain a purpose beyond helping humans understand the code and actually help enforce that the end result is reasonably bug free. Building code out of understandable pieces can ensure that the whole thing is reasonably understandable.
While the article is focused on “low level” bugs, the key insight is that human understanding has limits, and bugs show up when the human is overconfident in their understanding, and unwilling to verify the code by running it on a sample of test cases. This does seem to help explain why “high level” bugs occur, where the reality somehow doesn’t match the assumption. Maybe a dependency is on the wrong version, a global variable isn’t initialized in certain cases, etc.
In a world of LLMs producing code that isn’t always understandable to humans, it does raise the question of how to enforce that AI code is bug free, and maybe forcing coding agents to stick to human abstractions is the missing piece.