Pseudo-Mathematics and Financial Charlatanism

A 2014 paper in the Notices of the American Mathematical Society (american-mathematical-society, vol. 61 no. 5, p. 458) by david-h-bailey, jonathan-borwein, marcos-lopez-de-prado and qiji-jim-zhu. It is this wiki’s first peer-reviewed source and the first one that is not written by someone selling the thing it measures.

Its subject is backtest-overfitting: what happens to a backtest when the strategy being tested was chosen by searching over many configurations of itself. The authors’ claim is not that overfit strategies are common — it is that producing one is trivially easy, and that a backtest which omits how many configurations were tried carries no information about whether it happened.

The result

Searching N independent strategy configurations over a return series with no real edge still yields a best-of-N in-sample Sharpe ratio that grows with N. The authors derive the expected maximum and invert it into a Minimum Backtest Length — how much history you need before a given N stops manufacturing a given Sharpe from nothing. Two numbers from their Figure 2, both for skill-less strategies whose true out-of-sample Sharpe is zero:

• Seven independent configurations are enough to produce an expected in-sample Sharpe of 1 on a two-year backtest.
• Five years of data supports at most forty-five configurations before the same thing happens.

The working approximation they offer is MinBTL < 2·ln[N] / E[max_N]² years. Because it assumes the trials are independent, it is the conservative end — correlated trials need a dimension-reduction step (they suggest PCA) to recover the effective N.

They are explicit that this is a necessary and not a sufficient condition: a backtest longer than MinBTL can still be overfit. The bound tells you when a result is certainly uninformative, not when it is trustworthy.

Why it lands here

The paper’s practical demand is one line: report N. Their argument is that without it an investor cannot distinguish a strategy from the search that found it, and that journals and practitioners who publish backtests without the trial count are — the paper’s own framing — dressing a pseudo-mathematical argument in a validation it never received.

That is a direct hit on this corpus. Every performance figure in this wiki comes from a party that benefits from the number, and several arrive from an explicit search: strategy-optimization is a hyperparameter loop over a backtest, which is exactly the N the paper says must be disclosed and which none of the spoke’s engine sources report. chan-algorithmic-trading names data-snooping bias in its inventory of failures that survive correct code; this paper supplies the quantity Chan’s prose leaves open.

Related

backtest-overfitting · backtesting · strategy-optimization · algorithmic-trading · chan-algorithmic-trading