Validate a trading strategy without hindsight · 5 / 5
Freeze a holdout without leaking future outcomes
A file called holdout is not untouched if its outcomes influenced the strategy. Track when every input and trade outcome became available, not only which month labels the row.
Athenum8 minUpdated:
Freeze both the rule and the evaluation protocol
Use January–March for development, April for selection and May–June for final evaluation in this hypothetical schedule. At the end of April, freeze rules, universe, parameters, costs, execution assumptions and acceptance criteria. A moving average may legitimately use earlier historical prices; the restriction is future information, not a requirement to forget the past.
A development trade opened on 31 March but closed on 2 April has no realised outcome available at the March boundary. Fitting with that outcome and evaluating from 1 April as if the model were already frozen leaks information. Predefine a policy to remove overlapping outcome labels from the fit or delay evaluation until they are known. The exclusion depends on the actual outcome horizon.
Preprocessing has an information timestamp too
A learned scaler, feature selection procedure or missing-value model must be fitted only on allowed fitting observations. Computing its parameters from the entire January–June sample before splitting can import future distribution information into earlier decisions. Apply frozen transformations to later observations. A deterministic conversion by a known constant is a different operation.
After inspecting holdout results, any revision starts a new research cycle. If May influenced a changed stop, May is development or selection data for that revised rule. June can only be a new shorter holdout if it has not influenced decisions directly or indirectly. Otherwise gather a genuinely new evaluation period.
A positive combined result does not rescue a losing holdout
Suppose development earns +12R, selection +3R and final evaluation −2R after the declared costs. The combined total is +13R, but the holdout result is still −2R. Report the holdout separately with trade count, conditions and uncertainty. This sample does not support the planned positive-performance claim.
At a freeze time of 30 April close, historical prices and already completed April outcomes may inform the selected model. A trade opened on 30 April and closed on 2 May is not yet a known realised outcome. Its later result cannot quietly enter the earlier fit.
| Stage | Period | Net total | Role |
|---|---|---|---|
| Development | January–March | +12R | Build candidate rules |
| Selection | April | +3R | Choose and freeze a rule |
| Holdout | May–June | −2R | Evaluate the frozen rule |
- Development
- 12 R
- Selection
- 3 R
- Holdout
- -2 R
An arbitrary gap may not remove outcome overlap
Skipping one bar around a split does not automatically remove leakage from a trade whose exit label spans several days. Audit the availability of the actual target or outcome used in fitting. Do not choose the exclusion length after seeing which setting makes the holdout profitable.
Before acting
- Record a freeze timestamp and acceptance criteria.
- Audit outcome-label availability at every split.
- Fit learned preprocessing on permitted data only.
- Track every inspection and rule revision.
- Report failed holdouts separately from development totals.
Check your understanding
A researcher fits a scaler's mean and variance on January–June, then trains on January–March and tests on May–June. Does splitting after scaling preserve an untouched holdout?
Show the explained answer
No. Future distribution information already entered the scaler. Fit learned preprocessing on the permitted fitting data, freeze it and apply it to later observations. Then rerun the evaluation under a protocol that records which data have already been inspected.