Comment by fwlr

I deeply appreciate that the author went to the effort of faithfully and charitably summarizing the superintelligence claim before arguing against worrying about it. You only really need the first four premises (proof of concept, no quantum shenanigans, many possible minds, plenty of room at the top) for some broad arguments about long-term risks of AI, but for the kind of stuff that captures the imagination and the debate (hard takeoff) you do need the next two premises (computer-like timescale, recursive self-improvement). So far, so good.

I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though.

“Woolly definitions of intelligence” is just plain confused; first it takes issue with the assumption that intelligence is even a quantity, that could be measured like CPU speed - and then immediately goes on to theorize that “human-level” intelligence is a local optimum due to trade offs, and that “significantly smarter” entities might necessarily suffer from existential despair. But you can’t have “similar levels” or “significantly smarter” or “emulating lesser intelligences” if intelligence is not a quantity that can be measured!

The arguments from various famous physicists’ cats argue that intelligence alone may not give the ability to persuade lesser intelligences, and brute force is a less favourable matchup for AIs. My counter-argument would be there’s other ways besides persuasive communication and brute force; you could put cat food in the cat carrier and the cat would go in quite happily, and I suspect both Hawking and Einstein would have used their intelligence to figure this out pretty quickly. (For humans, offer them money.) The argument from emus seems to be mostly a way to include a fun anecdote about the Emu War.

The argument from pessimism doesn’t do much for me personally; ex diffido quodlibet, you can believe anything is impossible if you want.

The argument from complex motivations is crucially flawed; this is probably the only point where I seriously take issue with the author and think they’ve made a serious mistake. They summarize the orthogonality thesis as claiming “complex beings can have simple motivations” and then disagree, saying they think complex beings are likely to have complex motivations. Firstly this is not even a disagreement; “it is possible for a complex intelligence to have a simple goal” coexists perfectly well with “complex intelligences are likely to have complex goals”. Secondly, the author really needs to read Basic AI Drives, since that is actually where most of the “you are made of atoms that could be used for something else” argument comes from, and it makes a nearly air-tight case that regardless of motivation complexity or goals, any sufficiently intelligent agent will exhibit certain basic drives like securing resources and protecting its existence. https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...

The argument from Actual AI is fine, you could add epicycles to the AI doom arguments to account for the particularities of machine learning but it’s correct to point out that we really don’t see recursive self-improvement. I mean, OpenAI’s GPT basically has yearly release cycles. Credit to the author on this one.

The argument from the lazy roommate is just the complex motivations / orthogonality thesis argument rehashed, except contradictory since now the author is postulating a complex intelligence will have simple goals.

The argument from brain surgery is just the argument from Actual AI rehashed as well, making the (correct) point that we don’t see recursive self-improvement, and recursive self-improvement is the load-bearing premise in hard takeoff arguments.

The argument from childhood seems to be vaguely disagreeing with premise 5 (computer-like time scales). I don’t think it makes anything like a strong case against that premise, and with GPT we have pretty decent examples that computer-like time scales are the correct time scales, you can scale the training of models worth worrying about just like we expected, etc.

The argument from Gilligan’s Island is sound, but seems completely inapplicable. Yes, a super-intelligent chip designer stranded on a desert island is out of luck, but it’s not like we’re building AGI on the moon. We don’t even have moats around the data centers (although I think OpenAI has proposed that). What we are actually doing is, pretty much the moment we thought we might have something smart, we connected it to the internet and gave it every autonomous capability we could think of. People are literally right now wiring up ChatGPT to bank accounts so it can participate in e-commerce autonomously. That’s about as far from stranding it on a desert island as you can get.

Basically, this list of arguments (purportedly “against the substance” of superintelligence arguments, no less!) are variously unsound, unserious, or irrelevant, and the article suffers deeply from including them.

The next section, starting with “the Outside View”, is much better. This comment is already far too long for me to go into detail about each point, so I’ll summarize by saying I completely agree that worrying about AI doom does seem to bring along lot of weird unsightly behaviors, and if you don’t want to look and sound weird, you should stop worrying about artificial superintelligence destroying the world.