All that Glitters Is Not Gold — 888 backtests against their own out-of-sample record

Thomas Wiecki, Andrew Campbell, Justin Lent and Jessica Stauth, all of quantopian. SSRN 2745220, 9 March 2016; later in The Journal of Investing 25(3):69–80. Found by the research pass for growth edge 1 (a source that reports its N); read from the SSRN PDF.

The dataset nobody else could assemble

Quantopian could see something a journal cannot: the timestamp of every algorithm's code. The period before deployment is by construction the in-sample window the author had access to; the simulated performance accumulated after deployment is "true out-of-sample data." From 7,152 algorithms the authors filtered duplicates, forum clones and degenerate strategies down to 888 US-equity algorithms, backtested 2010–2015 with at least six months of genuine out-of-sample trading. The site's terms forbid reading the source, so every feature is built from returns, positions and transactions.

What the backtest predicts

• In-sample Sharpe predicts out-of-sample Sharpe at R² = 0.02 (Pearson, p < 0.0001), the abstract's "R² < 0.025". Annual returns are weakly negatively correlated across the two periods.
• Risk metrics carry over where return metrics do not: annual volatility R² = 0.67, maximum drawdown R² = 0.34 between in-sample and out-of-sample.
• More backtesting, larger shortfall. The authors log-transformed the total number of days a strategy was backtested during development and found a weak (Spearman R² = 0.017) but highly significant positive correlation with Sharpe shortfall (in-sample minus out-of-sample). That is the first measured instance in this spoke of the effect deflated-sharpe-ratio derives.
• A random-forest regressor over behavioural features predicts out-of-sample Sharpe at R² = 0.17 on a 20% hold-out (0.18 in cross-validation). Ranking by its predictions, an equal-weighted top-10 portfolio beat 99% of 1,000 random portfolios (Sharpe 1.8); ranking by in-sample Sharpe beat 92.16% (Sharpe 0.7), which the authors call reasonable but not significant.

Where it sits on the edge

Edge 1 asks for a source that states its trial count. This paper states its cohort N (888 of 7,152) and, more usefully, measures search intensity per strategy as backtest-days — a proxy for the configuration count rather than the count itself. So it is the third half-close on that edge, and a different kind from the two before it: deflated-sharpe-ratio supplies the correction, thousands-of-alpha-tests supplies N as a fund count, and this supplies the empirical slope — shortfall rises with the log of how long a quant kept testing. Nothing in the paper reports how many distinct configurations any one strategy tried, so the original ask stays open.

Caveats the paper carries

The cohort is one platform's hobbyist and semi-professional users, 2010–2015, US equities only. The out-of-sample window is short (6–12 months), the filters removed 88% of the raw sample and the authors say they "experimented with various preprocessing parameters" before settling on them — a selection step of their own that is disclosed but not counted. Random-forest results "were not always identical across different folds." The authors are the platform's employees and the paper argues for the platform's allocation model. None of that is hidden; all of it bounds the numbers.

Tier T1 — peer-reviewed (Journal of Investing), unique proprietary data, full methods. The interest is disclosed on the page.

Related

backtest-overfitting · deflated-sharpe-ratio · thousands-of-alpha-tests · strategy-optimization · backtesting · quantopian · marcos-lopez-de-prado