Gradient-boosted trees

Fit a shallow decision tree to the data, look at what it got wrong, fit the next tree to that residual, and keep adding trees — each one a small correction weighted into the sum. The trees are weak on their own and the ensemble is not. Compared with bagging (random forests), where trees are grown independently and averaged, boosting is sequential and each step is aimed at the current error.

This is the default method for tabular data, and the corpus can now say so with sources rather than by reputation.

What the evidence says

• It took over because an implementation made it cheap. The algorithm predates XGBoost; the system is what changed practice — 17 of 29 Kaggle competition winners in 2015, every top-10 KDDCup 2015 team, and the same accuracy as scikit-learn’s implementation at roughly 40× the speed on Higgs-1M.
• It still wins against deep learning on tabular problems. Grinsztajn et al. benchmark 45 datasets with ~20,000 compute hours of tuning per learner and find tree ensembles state-of-the-art at medium size (~10K samples), before counting their speed advantage.
• The reason is inductive bias, not maturity. Trees ignore uninformative columns, split on individual features (so they respect an orientation that rotation-invariant networks destroy), and fit irregular functions with jumps. Those are properties of real tabular columns, not of an immature competitor.

Where it sits in this spoke

classical-ml-algorithms holds the pre-deep vocabulary; this is the member of that family that never got displaced. It is also the missing control in two live arguments: open question 7 wants a time-series foundation model measured against a gradient-boosted tabular baseline rather than against seasonal-naive (timesfm-2-5-forecasting-tutorial), and the corpus’s broader demo→production thread (demo-to-production-gap) keeps finding that the deciding variable is data quality — which is exactly the regime where boosted trees are hard to beat.

Not covered here: LightGBM and CatBoost, whose differences from XGBoost — leaf-wise growth, ordered target statistics for categoricals — are real and unsourced in this corpus.