Comment by hwillis

> Except this AI doesn't exist. You're imagining there is such a technology, and this technology will just be able to see all the detail in the world it needs to learn

Obviously I'm imagining it. Strong AI does not yet exist. It's also obvious that it could exist, because humans do it. I'm only making two logical inferences here:

1: Future superhuman AI will have at least the capabilities of the human brain, because we know those abilities are possible, because we do them.

2. A future superhuman AI will have the ability to examine itself in memory and identify things about itself in a way that far exceeds human introspection: we can barely examine out own emotions, much less the actual neuronal contents of our heads.

> And it's also why I think he paperclip maximizer and gray goo scenarios are silly. Maybe it's theoretically possible to create something that would eat the world, but in order to do so, it would have to overcome every obstacle the world throws at it. [...] If you think deep learning is such a technology, then ask yourself to what extent ANNs understand themselves and you'll see they don't at all.

Well first off, they're quite good at it[1], but more importantly that's weak AI rather than strong AI. Arguing that weak AI is unlikely to be superhuman is plausible, but strong AI is definitely self-improving.

What I think you're saying is that you're skeptical of us being able to create strong AI out of current techniques, which is also reasonable. NN are not gonna evolve into skynet any time soon. But believing them to be categorically impossible requires the human brain to be special in some way- either beyond human comprehension or comprising a supernatural component.

[1]: https://en.wikipedia.org/wiki/Generative_adversarial_network