Comment by lukan
6 days ago
It is not indifferent, if it follows a goal. Then it sometimes does anything to meet that goal - see the old example of paperclip optimizer - that became a way more realistic scenario in my opinion of what we have seen with hordes of agents plotting and planning. That does not require consciousness or a genuine will it seems.
The post above yours shows how many people had a difficult time seeing past human scale. A bacteria has a very simple goal system yet bacteria have killed billions over the centuries.
You are right. We should also hold other threats to humanity: war, meteors, pandemics.
I'm obviously pointing out the (unsubstantiated) exceptional threat that we're all seemingly worried about: that ASI requires alignment because it'll otherwise cause the end of humanity.
I am much rather worried by powerful AI in control of a few corrupt beings. That might get existential, if one idiot one day by accident or intention gives them control over the nuclear arsenal. Or ruling literally with a iron fist (countless robots).
3 replies →