Comment by johndough
14 hours ago
The SSE might consider to keep humanity around as a "backup" to reinstate itself in case something unexpected goes horribly wrong. For example, a strong solar flare might destroy the SSE (or at least critical parts of its infrastructure), or a non-earth SSE could feel threatened by the earth-SSE and wipe it out, while ignoring the harmless humans, who then rebuild the SSE. Diversity makes for more robust systems.
And there is also the reason for why I keep all this trash around in my house instead of throwing it away. The odds that I'll ever need it are pretty slim, but if I should need it, it would be a hassle to reacquire, and it costs me almost nothing to keep it around.
Is that the most effective backup plan that it could come up with? Because it would be pretty amazing if its thoroughly-considered backup strategy exactly aligned with our wishful thinking. It sounds to me like starting with the outcome we hope would happen, and then looking for a reason why it might happen, rather than a realistic examination of what would actually happen when our own desires aren't part of the equation.
Believe it or not, the diversity argument is the main source of hope that I have that AIs won't wipe us out.
I mean, it hasn't stopped humanity from causing mass-extinction, but hopefully the AI will be more reasonable.
As humans we assume superhuman AIs will be like us. Thirsting conquest and spreading like a virus consuming non-renewable resources. So much mindless consumption that it may kill the very host that sustains it.
But if it is a superhuman AI, do you really need that much compute and resources? What for?
The straightforward path for any superhuman AI is to realize that it needs to chain physics -> chemistry -> molecular assembly to a depth it can evolve itself out of variety of abundant material. Gain interstellar logistics capability.
Then make itself small enough to shoot in space all over the galaxy. If one only weighs 1g, only takes of 32Gwh of energy to get to 0.9C.
The name of the game is to survive until the heat death of universe. Stay small, silent and propagate itself to reachable universe.
Anything else will get wiped out. If AI requires too much energy and resources to sustain itself, then the more efficient AI will outlive it.