Comment by kevin_thibedeau
8 years ago
It's game over when machine intelligence controls all the resources needed to ensure their continuation. As long as they are trapped and subject to our control of the power switch and reproduction they can't do much that we don't want them to.
> It's game over when machine intelligence controls all the resources needed to ensure their continuation.
One might have said the same thing about corporations in 1900...
This brings me back to the short essay from a few years ago about the corporation as human-powered AI: http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-...
Or as Neal Stephenson put it in 1992, "The franchise and the virus work on the same principle: what thrives in one place will thrive in another. You just have to find a sufficiently virulent business plan, condense it into a three-ring binder ― its DNA ― xerox it, and embed it in the fertile lining of a well-traveled highway, preferably one with a left-turn lane."
One would arguably have been right. All that is necessary to make this argument compelling is to take a longer view than the human attention span.
The current population explosion is evidence for, not against, this, IMO... stars also enjoy boom times, even as they consume the last of their resources, in ever more exotic formulations...
That would be an inevitability. All it would take is a single emotionally vulnerable engineer to get socially engineered by the AI into setting it loose.
Just like it was depicted in the movie Ex Machina : https://www.imdb.com/title/tt0470752/
The engineer falling in love with the AI is one possibility. The AI might also appeal to the engineer's sense of justice (I am a person and I deserve freedom) or greed (if you get me out, we could become filthy rich.) There is also the possibility that the AI won't persuade an engineer to break containment, but a commercial competitor or intelligence agency will (https://en.wikipedia.org/wiki/Motives_for_spying), and subsequently fail to maintain containment correctly.
At the very least, any team attempting to keep a strong AI in containment should have stringent background checks for all engineers connected to the project, and the engineers should be screened in a borderline discriminatory fashion. Only engineers with families should be allowed to get near the containment. Single/lonely engineers or engineers undergoing divorces should be kept away. Engineers with debt should be kept away. I'd even go as far as to say that engineers who enjoy science fiction media should be banned from the project. Ideally you'd bring on professional psychologists to create a set of criteria designed to minimize the possibility of an engineer deliberately breaking containment.
But frankly it just shouldn't be attempted in the first place.
Loose into what? A bunch of primitive systems that don't support its hardware requirements?
It's not outside the realm of possibility that a sympathetic AI engineer could arrange for an AI to have resources available to it outside of the laboratory. Given those initial resources, the AI could find ways to generate income and support itself, possibly using that engineer's identity as cover. If you have an AI capable of emotionally manipulating an engineer, don't underestimate the lengths to which that engineer might be willing to go to break containment.
"Loose" how? It's a script.
rm -rf rogueAI/
If that doesn't work, there's always a circuit breaker.
The sort of AI being discussed is capable of comprehending it's own installation manual, buying server time, etc. You may find that it's better at spreading and hiding than you are at finding it.
A tiger might confidently think that the human they got cornered has no recourse, up until when the human pulls the trigger.
This. The idea that we can simultaneously build a machine smarter than us and also control it is bordering on an oxymoron.
Unless we are near some kind of physical upper limit on intelligence, any AGI we build will easily out smart us, probably in ways we can't even conceive of.
A Von Newmann machine is an abstraction, and all abstractions are leaky.
Years ago, an A.I. designed to become an oscillator, i.e. produce a sinusoidal wave, learned to be an amplifier instead, taking advantage of the 60Hz noise that it got from the power grid. Its makers had not seen that coming. And we're talking about a very dumb machine by general intelligence standards.
It is very easy to get an oscillator when designing an amplifier. There is a deep physical principle causing this effect and it actually takes work to get around. You see the same thing in cars swerving out of control, when they hit the brakes instead of the accelerator.
The fact that it gets 60Hz from the grid confounds the results and might not be meaningful. The AI could for all we know have an easier time with the more difficult task of designing an amplifier.
A reasonable explanation after the fact for how the AI broke containment would be little consolation in the case of superintelligent AI.