Backtest bias: validate a trading strategy without hindsight

Audit signal timing, ambiguous fills, confirmation filters, parameter searches and chronological holdouts with reproducible worked cases.

Course outline

1. 1

   Find when a pivot signal actually became available

   A pivot marker is drawn after the fact. Reconstruct when the signal became available and recalculate the trade: a hindsight 16R becomes a timely 0.81R.

   7 min

   In short: A strict pivot low is only a candidate until the two following completed bars close with higher lows. Store the candidate time and the confirmation time separately, and enter at the next executable price after confirmation. In the example, that entry gives 0.81R; a hindsight entry at the marked low of 100 gives 16R.

2. 2

   Handle a candle that touches both stop and target

   OHLC bars do not show whether the stop or target came first. Keep both orderings: the same ten trades produce net bounds from 0R to 6R.

   7 min

   In short: An OHLC candle that touches both stop and target does not show which was hit first, so report the result under each possible ordering. These bounds are not probabilities, and their midpoint is not expected profit. In the example, ten trades with two ambiguous candles have net bounds of 0R and +6R.

3. 3

   Measure what an extra trading confirmation adds

   Compare a fixed signal with and without a filter. Keep rejected opportunities and costs instead of counting correlated indicators as independent votes.

   7 min

   In short: Judge an extra confirmation by its incremental effect: the same policy with and without the filter, on a frozen opportunity set, after costs. Agreeing indicators are not automatically independent evidence. In the example, the filter keeps +4R gross but nets +3.2R versus +2R at 0.1R cost per trade, from 8 trades instead of 20.

4. 4

   Keep the losing trials in a parameter search

   Audit the full search behind a winning backtest. Separate robustness scenarios, selected parameters and independent evidence to avoid overfitting.

   7 min

   In short: The best of many parameter trials is selected for being extreme, so its result is weak evidence of a persistent advantage. Record every trial with its settings, costs and result: in the example, 60 configurations give a best of +18R against a median of −2R. Trials on shared data are not independent tests.

5. 5

   Out-of-sample testing: freeze a holdout without leaking outcomes

   Freeze the rule and evaluation protocol before testing. Audit boundary-crossing trades and preprocessing so future outcomes never leak into the holdout.

   8 min

   In short: Freeze rules, parameters, costs and acceptance criteria before the holdout, and fit models and preprocessing only on data and trade outcomes already known at the freeze. Report the holdout separately: in the example, a combined +13R does not change a −2R holdout. Any revision after seeing it starts a new research cycle.

Open full-size diagram

1. Find when a pivot signal actually became available
2. Handle a candle that touches both stop and target
3. Measure what an extra trading confirmation adds
4. Keep the losing trials in a parameter search
5. Out-of-sample testing: freeze a holdout without leaking outcomes

Educational material. Examples do not establish a profitable strategy. Trading costs, gaps and liquidation can produce losses beyond a planned stop.