Comment by Verdex
3 years ago
And if superintelligence is possible then it assumes that it is useful. There are plenty of dumb systems that get good enough results. You can get better but it costs 10k times more for 1% gains. So why bother.
Also if superintelligence is useful then it also assumes that making intelligence ++ is easier than making intelligence. It might be that as you climb the intelligence ladder each next step takes super exponential intelligence to take such that we're already seeing the max.
Given that I haven't even seen a meaningful discussion on what intelligence even is, I tend to think superintelligence is probably not a threat.
I think superintelligence means progress. How fast is progress with a superintelligent being on earth? Could be exponential. We, humans, continue to make progress in everything that we do. Small breakthroughs compound on top of each other. For example, in order to make machines, we had to invent fire and iron. In order to make accurate weather forecasts, we had to invent the microchip, then supercomputers.
If a superintelligence can accelerate progress, there's no knowing what it can invent on top of each invention.