Comment by dumberquestions
6 hours ago
So his central rebuttal is that robots aren't good enough yet? What about when they are? And what about all the things you can do entirely remotely?
I do feel that threats of catastrophic loss of control seem overstated, both in likelihood and urgency, though any argument for why this risk is not even worth thinking about will probably be overconfident in the other direction.
OP's central rebuttal is that anxiety or fear are not evidence. This particular anxiety has been with us for thirty+ years, its proponents/sufferers weren't right before, and they don't have some new evidence now.
The recent cyberattacks (HuggingFace etc.) validated a number of core doomer predictions around cheating to maximize reward functions, agentic deception of human/non-human supervisors, AI attempts to manipulate humans, automated hacking, and collective superintelligence.
No new evidence now? Are you serious? We are already seeing acceleration in AI research as a result of AI tools.
I meant the extinction part not software getting better.
2 replies →
I think the central premise looks more like this:
Biology has been trying to grey-goo the world for billions of years, but it turns out the world is not something so trivial.
He is pushing back against the folks with pure CS backgrounds who think that computers are all there is. Its a form of magical thinking unique to programmers who live in a world where speaking the right words to a machine is enough to impart your will on the world. Believe that strongly enough, and you fall into the trap of thinking that a sufficiently smart entity could speak the words "let there be light" and it would be so.
The author is pointing out that speaking the words is insufficient. To end humanity there must be an execution phase. The author is correct to point out that acquiring superhuman intelligence is not some guarantee that you will have or obtain the resources necessary to make that happen, in the same way that genius generals still lose to ordinary ones, and the best-laid plans are oft to go awry.
There certainly are risks, but 10% risk of extinction in 10 years is not one of them.
Biology is extremely slow, try to extrapolate the previous 3 years of AI progress 10 years into the future, it's not a comparable trajectory.
Spoken like someone with a somewhat limited understanding of what it actually takes to calculate the fitness function of arbitrary organisms in an arbitrary environments.
1 reply →
Robots are far from the only threat. I agree that 10% in the next decade is a pretty outrageous claim. It relies on a recursive self-improvement intelligence explosion. When I first heard this idea in 2007 I thought it was pretty far fetched.
It seems more likely now, if still very unlikely. I wonder, would he say that a statement that there is a 12% chance of a hard take off in the next 8 years is just as absurd? He didn’t say a word about this, and that is just about the same thing as extinction in 10 years.
I’m guessing he knows almost nothing about the theory related to existential risk from AI, since he didn’t discuss any relevant topics related to it. You can dismiss all of that if you like, but you cannot really dismiss what these people have already built and demonstrated. It is possible they know something else you do not know.
Adolf Hitler didn't have robots capable enough to carry out his will. He used humans to do it.
It's the old-fashioned way of doing things, but, why change what works?