Comment by p-e-w
3 years ago
Apart from the fact that the entire article/talk comes across as incredibly patronizing, I'm having a hard time figuring out what exactly the author is arguing for or against.
Either AGI is an existential threat to humanity or it is not. But under no circumstances are social arguments like "What kind of person does sincerely believing this stuff turn you into?" relevant to that assessment. Nobody asks such questions about climate change, because most smart people nowadays agree that climate change is real and is a real problem. Sidestepping the essential issues with ad hominem ridicule of the proponents of AI extinction scenarios has the same intellectual flavor as climate change denialism.
Is superintelligence possible in the foreseeable future, and is it a danger to mankind? Those are the only questions that matter. Religion, Elon Musk, the personality traits of AI researchers, and stupid memes are entirely irrelevant. Unless of course the real goal is just to get clicks and attention.
> Nobody asks such questions about climate change
They do actually. Climate doomers are not a pretty group. They do shit like glue themselves to roads and try to vandalize paintings because they're convinced (wrongly) that the world is ending. The outside view here is that these people are weird, dangerous and cultlike and you don't want to get involved with them.
So listening to those signals can actually work. That's the message of Zion Lights, who joined Extinction Rebellion and eventually quit, telling the world it isn't a green movement but rather a cult oriented around its leader Roger Hallam.
https://www.thefp.com/p/climate-activism-has-a-cult-problem
The people I worked with had big hearts and good intentions. Some are still my friends.
But there were red flags.
At my first XR media training, I was instructed to cry on television. “People need to see crying mothers,” Jamie Kelsey-Fry, the trainer and longtime XR activist, told me. “They need to be woken up to what they should really care about.”
What she calls red flags is the outside view.
and somewhat ironically the answer is trivial. yes. just as if we were to start building nukes and irradiated the whole surface and most of the atmosphere too. could it happen? sure. it takes one powerful rouge state. how likely it is? unlikely.
now superitelligence is tens of orders of magnitude more complex than this simple rouge-geocidal-state case, but the math is the same. could it happen somehow? yes. how likely it is? well, that's the real question.
... and it seems Elizier thinks it's too likely. and in a sense he's right to act as the most doomy-gloomy person (because he thinks this directly helps decrease this likelihood). and it makes sense, by forcing the debate as much as possible it gives some chance to sufficient coordination between stakeholders to take this hazard somewhat seriously.