Comment by GuB-42
3 years ago
That's actually what I meant: "we don't know". But it seems that many people think that superintelligent AIs will enter a positive feedback of improvement getting to something incomprehensible, something the article is an argument against BTW.
But the naive interpretation of this leads to a singularity, and the fact it leads to a singularity shows that that interpretation is wrong and that there should be a limiting factor somewhere, the article suggests some of them.
As for the black hole, it is hard to argue using that analogy since you don't want to get close to a black hole for very well known reasons unrelated to the singularity, but let's get into sci-fi mode and say you could, it may be risky but it may also be worth it. The thing is: from the point of view of the travailler, you never cross the event horizon, and you get to know what's inside the black hole as the event horizon recedes from you, maybe you will never get back, but maybe there are greater things in there. And we are already trapped anyways as we are going one way through time, just as in black holes, things go one way through space, so maybe it is a risk worth taking.
Continued increase in computing power and connectivity is possible - that seems to be plainly correct in the short term. Continued increases appear to be able to make big changes to the world - compare 1950 with 1990 with 2023 and the internet, online banking, online shopping, email, smartphones, TikTok, Self-driving cars, ChatGPT, all these rippling shockwaves on the way to the event horizon are already rocking our human boat. Saying "there can't really be a singularity ahead" or "there isn't really an event horizon we will experience crossing" seems to be missing the point. The bad bit is worse shockwaves, unpredictable shockwaves, which sink the boat - not the boat arriving a the "end" and finding Cthulhu. Arguing "Cthulhu can't be there" isn't as reassuring as you imply it is. If there's a futureshock which wipes out humans, saying "well that isn't the end because there is no end, so your model is wrong" won't be much consolation.
> "but maybe there are greater things in there."
Probably not though; of all the arrangements of matter and energy possible, most of them are hostile or indifferent to Humans. Very few are beneficial to humans. Even Venus and Mars which are roughly Earth-sized rocky planets with atmospheres which orbit near the Goldilocks zone of a yellow dwarf main-sequence star, are still incredibly hostile to humans and they're far closer to Earthlike than most things in Space.
Heh, forgive me if I find the metaphor of “let’s send all of humanity into a black hole because it might be great” less than comforting.
I think the most likely scenario where we make something much smarter than us is bad. I don’t agree with all of what Yudkowsky says, but “Nature is allowed to kill you” is something that I think not a lot of people grasp on a real intuitive level. We are totally allowed to invent technology that gets out of control and kills us all. It could have happened with climate change if the earth had stronger feedback. We’ve been lucky so far, but as humanity gains power, we need a corresponding increase in our ability to coordinate and be careful to match.