Why do tree-based models still outperform deep learning on tabular data?
Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux, 2022 (NeurIPS Datasets & Benchmarks). T1 — a benchmark paper with its raw results released, and the “one benchmark against a deep model” the spoke’s gradient-boosting edge asked for.
45 datasets selected for having the characteristics of genuine tabular data, with hyperparameter search budgeted at roughly 20,000 compute hours per learner so that the comparison is not a tuning artifact. The finding: tree-based models — XGBoost, random forests — “remain state-of-the-art on medium-sized data (~10K samples),” and that holds “even without accounting for their superior speed.”
The three inductive biases
The paper’s value is in explaining why, which makes the result predictive rather than anecdotal. Neural networks lose on tabular data because they:
- Are hurt by uninformative features — trees ignore a useless column; a dense network must learn to.
- Are rotationally invariant, and tabular data is not. Column identity carries meaning that a rotation destroys; trees split on individual features and so preserve orientation.
- Struggle with irregular target functions — the smoothness prior that helps on images and audio is wrong when a target jumps at a threshold.
What it settles, and what it doesn’t
It settles the default: for a tabular problem of ordinary size, boosted trees first. It does not claim deep models are unusable — the boundary is drawn at dataset size and data character, and the authors publish the benchmark precisely so tabular-specific architectures can be tested against it.
For this spoke, it sharpens open question 7 rather than answering it. The tabular baseline that a time-series foundation model would have to beat is now identified and characterized; nobody here has run that comparison, and timesfm-2-5-forecasting-tutorial still stands as a claim measured against seasonal-naive on synthetic series. It also connects to data-cascades and pervasive-label-errors: the reasons trees win are properties of messy real columns, which is the same terrain those two papers say determines whether a system works.