Comment by YeGoblynQueenne

All that is reasonable, but I have asked both "when" and "how", above. If we don't know "how", now, then "when" becomes the crucial question. That's because if superintelligent AI is 10k years away, then it might as well be impossible, because we have no idea whether we will still have the same technological capability, or social structures, as in the current day, in 10k years. Also any action we take now to avert AGI, or control it, or align it, or anything, will be pointless because forgotten much sooner than 10k years.

I'm not talking about global collapse, btw. I'm mainly expecting that scientific advances in the next couple hundred years will leap-frog today's unscientific research into artificial intelligence. I'm guessing that we will eventually understand intelligence and its relation to computation and that we will find out that today's ideas about artificial intelligence never made any sense, nor had any chance of leading to artificial intelligence, of any sort.

You see, I trust science. And it's obvious to me that the current dominant paradigm of AI research is not science. So I don't believe for a second that, that paradigm, can really achieve anything approaching intelligence, running on a digital computer. Because that sounds like a very hard thing, and the kind of very hard thing we can only do with science.