Comment by mattmanser
3 years ago
Personally I find people with the superintelligence fetish frustratingly dumb too.
Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.
And that's before we even think about whether it would even have the motivation.
I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.
> Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.
When the "people with the superintelligence fetish" worry about AI, at least they provide a reason why it's likely that AI will make humans extinct.
What is your heuristic to determine what is likely?
The plateau of human intelligence isn't especially relevant given we are the product of evolution rather than engineering.
Personally,I think we stand a pretty decent chance of making a horrible mess of this all well before reaching AGI. However, a machine that could improve itself would not be limited in anyway that humans are; and I think we know little of how it would behave.
That is assuming it can't be controlled; otherwise it will behave as directed by whoever controls it.