Comment by strange_quark
14 hours ago
> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.
?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.
Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
>Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
That's the danger with creating a super intelligence, that you won't understand it. My dog is pretty smart, has good instincts and I'm quite confident he can survive in nature all by himself. But if a human decided to harm him, he doesn't stand a chance.
> That's the danger with creating a super intelligence, that you won't understand it
This argument is sophistry because it assumes the conclusion in its premise. It works backwards from extinction by proposing a hypothetical computer program that definitionally cannot be understood, predicted or controlled.
A substantive argument would propose reasoning that explains why we should believe such a program could be created.
No, that is not how reasoning works. It is perfectly valid reasoning to begin by positing entities that we may not be possible to create, such as supersonic airplanes, coherent light, quantum computers, rockets that can reach escape velocity, black holes, or infinitely long lines that are perfectly straight and infinitely thin. Indeed, you might say that the ability to reason about the consequences of premises that we do not know to be true is the thing that distinguishes reasoning, as such, from simple parroting.
It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved. This allows us to easily dismiss the most profoundly stupid arguments, such as, 'Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug?'
This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.
To be fair, he could probably fuck the human up pretty good if he had a mind to, too. Which may be the most likely reason that a human might decide to harm him: if he bites a couple of children, for example, or if a burglar wants to rob your house.
Seems like bioweapons (assuming "planet killing" means killing all the people, not literally blowing up Earth) are one way. Convincing the creator it would achieve their aim and could be contained when in fact it could not.
AI could invent something like social media. Obviously not social media because that's already been invented, but the next thing that slowly destroys society while seeming to be good.
> probably destroy almost anyone who survived the initial blast and fallout.
The difference between killing almost everyone, and killing everyone, is similar to the difference between almost getting pregnant and getting pregnant. A hundred thousand years later, a world where almost everyone had been killed would be teeming with humans.
> Can you explain how exactly an AI would develop a novel planet-killing weapon
I already answered that question in my previous comment, and the answer is that I am not going to explain how to develop weapons, but the larger point here is that we already can't explain how exactly AIs do innumerable things we see them doing every day, which is why we're having this conversation in the first place.
> I seriously don’t understand this argument
That is a point that I think everyone can agree on: you do not.
I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.
Life in theory could restart with our weapons, but what’s the point with the other scenario.
Being scared about novel AI weapons sounds like the people who were scared that turning on the LHC would create a black hole that would suck in the Earth.
The most important difference is that this time it's the people who know the most about the topic who are concerned, and the people who know the least about it who are blasé. (Also, Yann LeCun.) With the LHC it was the other way around.
8 replies →
> I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.
What a silly distinction to make.