Comment by jules

Well yeah, but then you're basically putting the meat of the NN algorithm into the SVM. I'd call the resulting algorithm a neural network with an SVM frosting. You might as well train naive bayes directly on the final nth layer of the NN instead of SVM on the (n-1)th layer, would be an almost equally weak argument for the thesis that NNs are not superior to the other algorithms on this task, since basically all the power is coming from the NN.