Comment by sobellian
16 hours ago
If anyone believes a sufficiently smart consortium of humans could exterminate 10% of humanity, that alone should be fixed by identifying the pathways through which it could be done. The existence of AI is not needed to be concerned about that possibility.
And dealing with those pathways individually would be far more productive than attempting to control the proliferation of algorithms that think, which in the long term is probably impossible. It also leaves us on far firmer scientific footing. An individual pathway - bioweapons, nuclear weapons, etc. - is far easier to reason about and accept/reject a notion of feasibility. Treating an ASI as a machine god and asserting futility in face of that god is not going to get anything done.
Part of the point in all this is that we can't predict what an ASI would do. All we can say for certain is that we would not be able to control it, because that follows a simple logic that between two entities with differing goals, the more intelligent entity is more likely to see their goals realized. It's interesting to me that you seem adamant that we identify and fix the pathways through which smart humans could exterminate 10% of humanity, but fail to see that preventing ASI is doing exactly that.
I understand the point, and I'm telling you this is just another form of Pascal's wager.