Comment by srean
2 hours ago
I think you should get less annoyed.
> It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple.
I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct
https://web.archive.org/web/20170428225156/http://www.cse.bu...
https://web.archive.org/web/20130412062821/http://cs.ecs.bay...
And I like the bayesian interpretation too. Murphy's "Probabilistic machine learning" has an occam's razor section.
The idea is that a complex model explains many more configurations (datasets) than a simple one. So its (prior) probability distribution is lower on the data seen (to compensate for the other possibilities it might explain). So the (marginal) likelihood that the simple model is correct is higher if it fits the data well enough.
[1] https://probml.github.io/pml-book/book1.html
This is a longstanding principle in model-fitting. More parameters, almost always, improves the ability of the model to fit to any particular data, in-sample. The model with the least parameters is both the simplest in principle and has the best chance of not overfitting.
True.
It so happens that one gets the best generalization error bounds when one combines PAC with Bayesian ideas -- the PAC-Bayesian bounds.
The notion of "simplicity" can be completely arbitrary, though. It's enough that there are only finitely many hypotheses simpler than the best hypothesis (assuming there's such a thing as a best hypothesis). So as you eliminate hypotheses incompatible with the data, at some point you'll have eliminated all simpler hypotheses, and the simplest hypothesis left will also be the best hypothesis. If simpler hypotheses are also more likely to be correct, you get there faster, but it's not required.
Except for the fact that eventually we are all dead. So it is kind of important to get there faster.
For complicated hypotheses, where complicated is defined appropriately, it takes many many examples to realize that it was a wrong hypothesis all along. There lies the rub.
For a particular instance of a learning problem we can't tell much, however using a Occams razor over many instances, one would be correct more often than not. Provided, of course, the PAC assumptions are true or they are not very far from being true.
How far is not very far ? That gets very hairy to quantify.
It does get you there faster. The overwhelming majority of incorrect models are more complicated than necessary.
Removing them at effectively zero cost is extremely valuable.
You choice of basis matters, eg, wavelet versus sinusoid.
None of your links work for me.
Ah! from my very dated and messy bibtex file comments. Wait, let me search for them on archive.org.
Fixed.
There are also various metaphysical theories that posit that the universe is algorithmically generated in some sense or the other, and from many of those theories it follows that simplicity is a fundamental feature of reality, which yields an even stronger version of Occam’s Razor.