Comment by a2ff6eeb0
4 hours ago
If we get effective AI, humans aren't going to remain in charge -- that much is inevitable. If you had a governance machine that made your country perform 10% better than other countries, so long as you deferred all decisions to it, you'd defer those decisions or get left behind. And AI probably has the potential to do far better than 10%.
Either everyone, globally, gives up on AI development, or AI is going to be in charge of all human development, and we'll just be along for the ride.
Right now, the work ahead is to shape AI so that the decision making it does as it carries us along is something we like.
No, because if this were the case, the UK wouldn’t have royals, North Korean wouldn’t be a dictatorship, China wouldn’t have the CCP, and America wouldn’t be a two party state.
Hardly any existing polity would willingly hand over control to an AI.
More likely an AI would overwhelm human polities through endless hacking and warfare.
You're assuming a handover happens in a single definable moment. More realistically it will be one small decision making power at a time, so that at any given moment it never really feels like control is being given up. Boiled frog.
I mostly agree, though there are versions of ASI that I believe leave humans “in charge”.
Most of my probability mass for good outcomes is around some variant of “guardian angels” or “benign machine god”. Banks’s Culture novels being on the realistic end of that range. “Keep humans around because the world is more interesting with them in it”.
But it’s possible that we might manage to construct some entity that really loves humans and wants to shepherd them while leaving them in control. (Perhaps declining to indulge the most self-destructive tendencies.) This feels like the very hardest alignment path to engineer.
The other branches of course are post-humanism; human uploads and “merging” with AI, whatever that means. Or post-humans augmented to have intelligence sufficient to eschew AI and take back their destiny.