Research note: five ways a backtest can mislead

A historical simulation answers a narrow question: how would this rule have behaved on this data, under these assumptions? The answer is easy to get wrong in ways that flatter the rule. These are the failure modes that shape our methodology. This note reports no results.

1. Look-ahead bias

Using information that was not yet available at the simulated time. The classic cases are subtle: a price series adjusted for later splits, fundamentals attached to the period they describe rather than the date they were published, or an index membership list taken from today.

Guard: every record carries an as-of timestamp; queries are bounded by it.

2. Survivorship bias

If the dataset contains only companies that still exist, failures have been silently removed and the universe looks healthier than it was.

Guard: keep delisted and discontinued instruments in the historical universe.

3. Data snooping and overfitting

Try enough variations of a rule and some will fit the past by luck. The more attempts, the more likely a good-looking result is noise — and if only the winner is reported, the evidence is inflated.

Guard: register every experiment, including failures; use out-of-sample and walk-forward validation; interpret results in light of how many were tried.

4. Unrealistic costs and liquidity

Ignoring spreads, fees, market impact, or the fact that a position cannot always be traded at the quoted price turns small apparent edges into large ones.

Guard: state cost and liquidity assumptions before the run; test sensitivity to them.

5. Model leakage (specific to language-model agents)

A model trained on text published after the simulated date can carry knowledge of what happened next. A simulation that uses such a model may be partially “remembering” the answer. This is hard to detect and hard to bound.

Guard: record model versions, prefer models whose training cutoff precedes the simulated period, and treat any other model-assisted historical result as suspect.

The common thread

None of these require bad intent; they arise naturally from convenient data and convenient workflows. That is why the controls are procedural — written down, registered, and reproducible — rather than left to individual care. See the methodology for the full set of rules.