Comment by idlewords

Fair comment.

The other thing that feels weird to me about the "argument from Slavic pessimism" is that it seems like an argument whose target is fundamentally different from that of everything else you're saying.

You begin by saying "Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity", and you say things like [note: these are paraphrases not quotes] "the very idea of superintelligence doesn't make much sense" and "no matter how smart a computer system, it will be fatally handicapped by being only a computer system rather than something physically embodied like us" and "humans quite often have difficulty imposing their wills on things less smart than themselves" and "superintelligent AI systems would have complex motivations rather than simple ones, and therefore literal paperclip-maximizers aren't a likely threat" and so forth.

All of these are indeed arguments against "superintelligence as a risk to humanity". But then, dropped into the middle, we have "if superintelligence turns out to be a threat, MIRI-style attempts at addressing that threat are a waste of time because humans are too stupid to do that sort of thing right first time". Which is addressing a completely different question from all the other arguments. It feels out of place.

This isn't a completely independent criticism from my previous one. It's unsurprising to me, where arguably it should have been surprising to you, that your argument-against-Yudkowsky ended up being something Yudkowsky basically agrees with -- because it's not really an argument against Yudkowsky, it's an argument that even if Yudkowsky is right we're probably still screwed. And I'm pretty sure that even in 2016 he would have said something like "we're probably doomed, but this is the approach that seems to me to give us the best chance of escaping doom".

(This being for some reason the sort of topic where people tend to label you as Friend or Enemy and take sides based on that, I should maybe lay my cards on the table. I agree that "intelligence" is a slippery notion but it seems obvious to me that there are plenty of ways in which something could be Smarter Than Us that would have broadly the kind of significance Bostrom, Yudkowsky et al think "superintelligence" would have. I suspect that for combinatorial-explosion-type reasons the actual benefit of being much smarter than us is smaller than B, Y, et al think. I suspect that we've already picked enough of the scientific/technological low-hanging fruit that the practical benefit of being much smarter than us, in terms of massive technological advantage, is smaller than B, Y, et al think. I suspect that nanotechnology in particular has much less potential than Yudkowsky at any rate tends to imagine. I suspect that the many cognitive flaws in human minds aren't "magically" exploitable and that even if they were learning how to do it would be difficult for an AI system. For all these reasons I am not convinced that superintelligent agent-y things would necessarily mean our doom, but I don't see any grounds for confidence that they wouldn't. It may turn out that we never figure out how to make things that are genuinely much smarter than us; it looks to me as if maybe we're one or two breakthroughs away, but who knows? Again, I don't see grounds for confidence that this won't happen. Today's LLMs are very impressive but it looks to me as if they are missing some fairly fundamental things that we don't currently have a plausible path to teach them (hence, "one or two breakthroughs" above). They pose plenty of less-exotic ethical issues already and those are predictably going to intensify even with "ordinary" development of the technology. I have never understood why so many people talk as if we have to either worry about near-term "mundane" AI issues like misinformation, job losses, exacerbating inequality, etc., or worry about longer-term exotic concerns like "will it literally kill us all?", as if worrying about one somehow guarantees that the other isn't a problem. I mention all of this not because I think my opinions are uniquely insightful but so that anyone who prefers to discount ideas heavily depending on "what side" the person whose ideas they are seems to be on can have enough information to decide what side they think I'm on :-).)