Backtesting

Backtesting is replaying a trading strategy over historical market data to estimate how it would have performed before risking real money. It is the validation step of algorithmic-trading, and the quality of a backtest hinges on one thing: whether the simulation faithfully reproduces what the strategy could actually have known at each moment.

Lookahead bias — the central hazard

The cardinal sin of backtesting is lookahead bias: letting a strategy use information from the future (a later bar, a not-yet-known close) when making a past decision. A backtest with lookahead shows fantasy returns that evaporate live. The event-driven architecture is the standard structural guard — data arrives as a forward-only stream of events, so the strategy can’t peek ahead. banbot frames its event-driven engine exactly this way; tensortrade faces the same hazard from the RL side, where the agent’s observer must not leak future bars into the state it sees.

The other failure: a correct backtest that still lies

Lookahead is a bug. chan-algorithmic-trading catalogues the failures that survive correct code, and it is the most specific inventory in this corpus:

Data-snooping bias — the strategy was selected by searching the same history it is scored on.
Survivorship bias — the delisted names are missing from the dataset.
Primary versus consolidated quotes, and the venue dependence of currency quotes — the price you backtested is not the price any single venue showed.
Short-sale constraints — the leg the strategy shorts may not have been borrowable.
Futures continuous-contract construction, and closing versus settlement prices — two arbitrary choices that silently change every futures result.
Regime shift, where “even the most correct backtest will fail to predict the future returns of a strategy” (market-regime-analysis).

Chan’s summary is that “the same theoretical strategy can result in spectacular profits and abysmal losses, depending on the details of implementation.” Six of the seven are properties of the data rather than the engine, which is why they survive an event-driven architecture and a perfect backtest/live parity story. The spoke’s engine-side sources (banbot, tensortrade, vectorized-backtesting) all address the seventh problem and none of these.

Where it sits in the validation ladder

Backtesting is the first of three stages: backtest on historical data (with input-optimization and risk hygiene — perturb inputs, Monte Carlo, model slippage and commissions), then forward-test on out-of-sample data, then live-test real fills against both algorithmic-trading-wikipedia. Forward testing is the explicit guard against over-fitting the backtest — the same hazard lookahead bias creates from the other direction.

How many times did you run it?

The ladder above is a procedure; pseudo-mathematics-financial-charlatanism supplies the number that decides whether any rung means anything. Selecting a strategy by searching N configurations of it inflates the best in-sample Sharpe on its own, with no edge present at all: seven independent configurations reach an expected Sharpe of 1 on a two-year backtest whose true out-of-sample Sharpe is zero, and five years of history supports only about forty-five before the same thing happens. The paper inverts this into a Minimum Backtest Length — the history a given N needs before its winner stops being an artifact of the search — and its practical demand is that a backtest report N at all. See backtest-overfitting. This is the one hazard on this page that a correct engine cannot fix, because it is a property of how the strategy was chosen rather than how it was simulated.

One codebase, two modes

A recurring goal is backtest/live parity — running the same strategy code in simulation and in production so a backtested result actually transfers. banbot makes this its headline design choice (single codebase for both), which also means the simulator’s fill/commission model has to match the live exchange closely enough to trust.

Performance

Backtesting is compute-heavy — a strategy may be replayed over years of bars across many symbols and timeframes, often inside a hyperparameter-optimization loop that runs the whole backtest hundreds of times. Hence banbot’s “1 year of klines in seconds” pitch: backtest speed caps how much strategy search you can afford — and which optimizer regime fits (cheap backtests favor population methods like cma-es; expensive ones favor sample-efficient BO).

Two engine families

The field’s catalogs sort backtest engines into event-driven and vector-based awesome-systematic-trading. The split is a direct consequence of the paragraph above: stepping through events costs speed and buys the anti-lookahead guarantee plus live parity; computing over whole arrays buys thousands of strategy variants per second and hands the lookahead problem back to the programmer. Which one is right depends on whether you’re sweeping a strategy space or validating a candidate you intend to trade.

When the author distrusts his own backtest

Koshtenko reports 64–65% accuracy and an encouraging profit curve from a home-built Python simulator, then says plainly he doesn’t trust it and would rather run the MQL5 Strategy Tester against real tick data. The distinction he’s drawing is the one this page is about: a simulator you wrote yourself inherits every assumption you made while writing it, including the ones you’d never write down — fill prices, spread, slippage, whether the bar you traded on had closed. A vendor tester on real ticks doesn’t remove those assumptions, it just stops them from being yours. Worth holding beside the ladder above: his system has been backtested and forward-tested and never traded live, which is exactly where the ladder says the interesting failures start.

A specimen arrives the same week as the warning (added 2026-08-03)

chan-algorithmic-trading landed on 2026-08-03 with data-snooping as its organizing worry. superior-skills landed hours later advertising a “validated” strategy with backtest evidence: Donchian Strong-Regime, BTC, 162 days — 6 trades, 100% win rate, 0% maximum drawdown, +6.69%.

Worth stating flatly, because the presentation is designed to read as strength. Six trades is not a sample. A 100% win rate over six trades is what you get roughly one time in sixty-four by coin flip, before any selection over strategies and parameters. And a 0% drawdown is not evidence of risk control; it is what a rule that almost never trades looks like. Every one of those three numbers is more suspicious the better it looks.

Contrast the same repo’s second strategy — 84 trades, four pairs, 65.5% win rate, 18.5% drawdown — which is a claim with enough events behind it to argue about. The lesson for reading this corpus: check the trade count before the return, and treat a perfect record as a report on sample size rather than on skill.

The vendor’s own disclosure is more careful than its table, naming the single 162-day window and declining to annualize. So the failure here isn’t concealment; it’s that “validated” got attached to a result that a walk-forward split couldn’t even be run on.

event-driven-trading · vectorized-backtesting · multithreaded-ml-trading-robot · superior-skills · agent-deployed-trading · supervised-learning-trading · algorithmic-trading · algorithmic-trading-wikipedia · chan-algorithmic-trading · mean-reversion · banbot · tensortrade · awesome-systematic-trading · synthesis