Comment by joe_the_user
8 years ago
Well, just about all the measures of potential future computer-intelligence more or less use "human abilities" as a placeholder rather than quantifying these abilities. That is indeed a testament to how little we have charted this intended final goal.
So basically, not only do we not have a road map, we don't know where we are going. That may be a reason for an extreme pessimism or it might be a reason for extreme uncertainty. Is adapting to the environment without prompting a small piece or a big piece? Could intelligence be a simple algorithm no one has put forward yet? If we don't know the nature of intelligence, we can't answer this sort of question with any certainty either way.
No one has put forward a broadly convincing road to intelligence. But maybe some of the so-far unconvincing roads could turn out to be right.
I think if we truly do create AI we will, to some extent, have stumbled into it, but that's OK. I think one of purposes of AI research should be to refine our definition of what intelligence is, because we don't have a very good one.
Perhaps, one day we'll "accidentally" figure out how to make it, and then we try to figure out what it is, because figuring out what it is in wetware hasn't been easy. Or maybe we'll figure out how to make it, but never really understand what it is.
It seems to me, that if we ever come up with AI that can exceed human intelligence (whatever we choose that to mean), that we might not ever be able to understand completely how it really works.
Even more interesting, if we were to achieve this AI, we also might not be able to make use it of it, because if it's truly "intelligent", then it will have a free will, and it might not wish to cooperate with us.