Comment by srean

Neural nets are quite a bit more than matrix multiplications, at least in their current representation.

There are layers upon layers of nonlinearity, be it with softmax or sigmoid. In the tangent kernel view it does linearize.