Comment by ethanbond
3 years ago
This seems quite presumptive. First, intelligence doesn’t seem to be unidimensional. A 140 IQ person can be fooled by an optical illusion just the same as anyone else. It’s just not a problem that’s able to be intelligenced away from our cognition. That doesn’t mean a 140 IQ person can’t beat an 80 IQ person in many many other competitions of intelligence.
Second, if you are truly “accepting the premise” of superintelligence, a superintelligence would know exactly this line of reasoning and just opt to at least mimic vulnerability to prompt injection.
I wouldn’t hang civilization on this proofpoint. Doesn’t seem meaningful at all.
> I wouldn’t hang civilization on this proofpoint.
And I wouldn't risk, or slow down getting to, the potential benefits of developing AI further, which are tangible and measurable, because of vague threat scenarios with little to no evidence or methods of measurement that seem like good plots for a SciFi B-Movie.
Pascals wager, as an argument, relies in no small part on the assumption that to believe in a god-like entity doesn't come at a significant cost. Slowing down or abandoning the development of AI however, does.
“I don’t know how to mitigate the risk without giving up something I like” is a completely different argument than “there isn’t risk.”
AI optimists tend to conflate the two for some reason.