Comment by hodgehog11
4 hours ago
Absolutely agree on both counts. Gradient boosting is the most commonly known and most successful variant, but it's the decision tree structure that is the underlying architecture there. Decision trees don't have the same "implicit training bias" phenomenon that neural networks have though, so all of this is just model bias in the classical statistical sense.
Can NNs be made to be better than trees on tabular data with some further constraints, or something?
You might find this interesting: https://arxiv.org/abs/1909.06312