Comment by andai

>Is superintelligence just a memetic hazard? [Overblown fear by smart people who are too easily convinced.]

Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)

If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.

Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.

Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/