GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation
Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong and Doyen
Sahoo — Salesforce AI Research (Xu Liu also NUS), arXiv 2410.10393, October 2024. Read from the
PDF with pypdf. T1 — a benchmark paper with a published leaderboard, a released pretraining
split and its full result tables, and the source open question 7 has been waiting for since it was
written.
The spoke has argued about time-series foundation models for a month on one tutorial that generated its own data (timesfm-2-5-forecasting-tutorial). This is the same question asked on 23 real datasets, 144,000 series and 177 million points, across seven domains, ten frequencies, univariate and multivariate inputs, short to long horizons — 97 dataset/frequency/horizon configurations in all, against 17 baselines.
The answer, and it is not the one either camp wanted
Aggregated over all 97 configurations, with every score normalised so Seasonal Naive = 1.00 and lower is better:
| Model | class | CRPS | avg. rank |
|---|---|---|---|
| PatchTST | deep, trained per dataset | 0.563 | 5.72 |
| Moirai-Large | foundation | 0.593 | 5.99 |
| TFT | deep | 0.596 | 6.76 |
| iTransformer | deep | 0.626 | 6.37 |
| Chronos-Large | foundation | 0.629 | 8.31 |
| TimesFM | foundation | 0.683 | 8.55 |
| Auto_ARIMA | statistical | 0.811 | 13.15 |
| Seasonal Naive | statistical | 1.00 | 16.64 |
| Auto_Theta | statistical | 1.41 | 15.21 |
The best model on this benchmark is a supervised transformer fitted per dataset, not a foundation model. PatchTST takes the top average score on every metric the paper reports; Moirai-Large is second. So the zero-shot claim survives against the classical baselines and loses to the thing it was supposed to replace — a model that does train on the target series.
The aggregate hides a cleaner split, which is the part worth carrying:
- By frequency. At the highest frequency (secondly) the leader is a statistical baseline, Auto_Theta. Minutely and hourly go to deep models (iTransformer, PatchTST, TFT). From daily to yearly, foundation models win outright.
- By variate count. Univariate is foundation-model territory (Moirai-Large). Multivariate goes to PatchTST and iTransformer, and Moirai is the only foundation model in the field that even accepts multivariate input.
- By domain. Foundation models lead most domains and lose Transport and Web/CloudOps — the two with the highest entropy, the weakest trend and the most lumpiness.
Read together: pretraining buys you the broad, slow, single-stream patterns. It does not buy you noise, and it does not buy you cross-series structure.
Two caveats the authors state themselves
Partial leakage. TimesFM, Chronos and Moirai all have "partial data leakage issues for GIFT-Eval" — their pretraining corpora overlap the evaluation sets. The paper's answer is a non-leaking pretraining split of its own (71 univariate + 17 multivariate datasets, 4.5 million series, ~230 billion points) and a leakage study in an appendix, but the headline foundation-model numbers above are still measured under the leak. They flatter the foundation models, and the foundation models still lose.
Whose models these are. Moirai is Salesforce's, and Gerald Woo is an author of both Moirai and this benchmark. That is the maker grading their own homework — except the paper's own conclusion is that a model Salesforce did not build wins, which is the direction bias does not usually run.
The paper also declines to support a scaling law: bigger foundation models are not reliably better here, and the pattern only holds in the energy domain and in univariate forecasting, on the ranking metric alone.
What it does not settle
There is no tree ensemble in this benchmark. Five statistical models (Naive, Seasonal Naive, Auto_ARIMA, Auto_ETS, Auto_Theta), eight deep models, four foundation-model families — and no LightGBM, no XGBoost, no boosted trees of any kind. The method a practitioner reaches for first on tabular data, and the one tabular-deep-learning-benchmark says is still state of the art at ~10K samples, is simply absent from a benchmark whose name begins with "General."
So open question 7 is half answered. The ARIMA/ETS half is settled on real series: foundation models beat the classical per-series fits comfortably. The gradient-boosted half is exactly as open as it was, and it is now open in a sharper way — the leading benchmark in the field does not run it.
Links
- timesfm-3 — claims the top foundation-model rank on this benchmark a year later, in a chart with no numbers, and against foundation models only. It does not contest PatchTST's win, and it is built at the multivariate corner this paper says pretraining did not buy.
- timesfm — evaluated here, and mid-table: rank 8.55, behind three deep models and two other foundation families.
- time-series-forecasting — the evaluation design this spoke wrote from a synthetic-data tutorial; GIFT-Eval is the same design at benchmark scale, using CRPS and a rank statistic over configurations rather than a single MASE.
- gradient-boosted-trees — the baseline that is missing from all 97 configurations.
- tabular-deep-learning-benchmark — the other benchmark in this spoke that budgeted its tuning honestly (~20,000 hours per learner) and reached the opposite conclusion about deep models, on tabular rather than sequential data.