Comment by hansvm

Interestingly, there exist problems which provably can't be learned via gradient descent for them.