Comment by narrator
8 years ago
I think we will find limited use for general AI since we won't be able to properly regulate or understand the behavior of machines that are smarter than us. Alpha Go Zero can't explain why it plays chess the way it does except for showing how it evaluates the board, but the reasoning is buried in millions of mathematical calculations accreted via millions of games of self play that we can't understand the intuition of. Of course, many smart players can't really tell you systematically how they make the moves they do either. AI will be like a dictator who makes his decisions by intuition only.
An intelligent black box can still be "useful". It could help cure diseases or get original insights in many fields for example. Even if we don't understand how it thinks.
Sure, but these are narrow AI applications. The great and powerful Oz general AI will be of limited utility unless there is a man behind the curtain.
If narrow AI can be useful despite being a black box, AGI can also be useful even if it's a black box.
All the human scientists in the world are black boxes. An AGI could be another researcher, just maybe more productive.
We don't need to understand how it thinks to understand the paper it writes.