Comment by pjin
13 years ago
To nitpick at the math: "No free lunch" results are asymptotic in the sense that they necessarily hold over the _entire_ domain of whatever problem you're trying to solve. Obviously, algorithms will and do perform differently over the relatively few inputs (compared to infinity...) that they actually encounter. It's similar to undecidability: just because a problem is generally undecidable doesn't mean you can't compute it for certain subsets of input, and compute it reasonably well (for some definition of reasonable).
Agreed... I was in a rush to catch the train this morning and I didn't have chance to elaborate, I shouldn't do that.
However, my point was that most of the algorithms used on that link (ANN, SVM, etc) had similar expressive power (VC dimension) and had been proved to have similar performance between them in object recognition.
People normally take advantage on their specific properties rather than paying too much attention how well the algorithm would perform (since either SVM and ANN are expected to perform reasonably well). I still maintain my opinion that any difference in classification performance is more likely to be related to how the team managed the data instead of the chosen algorithm.
Deep convolutional learning is the difference here and indeed seems to be an interesting architecture which the current state of the art only support ANN. But that doesn't mean that somebody wouldn't come up with a strategy for deep learning on SVM or another classification technique in the future.
Although SVMs and layered neural nets have similar expressivity, the similarity is very much like turing completeness. i.e. Can't tell aparts the haskells from the unlambdas. SVMs express certain functions in a manner that grows exponentially with input vs a deep learner which tends to be more compact. The key to being a deep learner is in using unsupervised learning to seed a hierarchy of learners learning ever more abstract representations.
Also, Multilayered Kernel learners already exist.
"The key to being a deep learner is in using unsupervised learning to seed ..."
Exactly! that was my whole point which doesn't makes sense now that the title had changed.
"Also, Multilayered Kernel learners already exist"
I didn't know that and I'll check that shortly, thanks for the info.
That's why they include which features they used, which is educational.