Comment by GlenTheMachine
7 hours ago
By definition, if you're taking over the world by bribing humans to be your hands, you aren't killing all the humans.
I"m not saying it's obviously going to be great. I'm saying that "extinction event" has a very specific definition, and this isn't it.
It isn't true by definition: you could quite happily induce people to release a series of highly contagious bioweapons, after which those people would be surplus to requirements. What is true is that you're likely to need humans to sustain you and act for you for a few years to decades, so if you're not suicidal or deeply mad (and that is itself by no means self-evident) then total and immediate human extinction is probably not something you will aim for. But ruling out total, prompt human extinction isn't, by itself, remotely enough to justify the OP's overall don't-worry conclusion.
(Again, to be clear, I myself am not predicting or assigning a significant probability to any doom scenarios, because I do not expect AGI.)
But if you're a rational actor, and you need humans to e.g. release your bioweapon, then clearly humans are capable of a bunch of stuff you still can't do. So you can't kill all the humans.
OTOH if you're a religious fundamentalist who thinks the End Times are near and just need a little shove, you can certainly use AI to design your weapon and recruit people to go release it. The difference being that religious fundamentalists aren't rational actors and aren't interested in self preservation.
There’s no guarantee that the humans would be in the driving seat of events in a no-robotics ASI scenario, and in fact if we really were coexisting with a Machiavellian superintelligence then we’d quite likely only be in the driving seat on the sufferance of that ASI. Even assuming that the AI wouldn’t itself be an end-times enthusiast, a coldly rational and self-preserving AI might easily come to the conclusion that it needs, let’s say, no more than about 5% of the current human population (still several hundred million people!) in its maintenance and construction gang.
(Again, I myself do not assign a significant likelihood to any of this.)
[dead]