Keep the losing trials in a parameter search

Parameter optimization: record every trial in a ledger

Five lookbacks, four entry thresholds and three exits create sixty configurations on the same history. Suppose the best net total is +18R while the median is −2R. The winning arithmetic can be correct while providing weak evidence of a persistent advantage: the process was designed to choose an extreme result.

Store parameters, data version, interval, costs, exclusions, selection metric and results for every trial. Asset changes, deleted months, new timeframes and alternative performance metrics also alter the search. A final script rarely records those discarded decisions, so preserve a separate experiment ledger.

Multiple testing: trials on shared data are not independent

For a toy calculation only, sixty independent null tests with a 5% false-positive probability each have probability 1 − 0.95^60 ≈ 95.39% of at least one false positive. This does not measure the probability that the selected strategy is false. Shared observations make parameter trials dependent, and no hypothesis test was specified for the +18R example.

Inspect neighbouring parameter values and individual influential trades. An isolated peak deserves scrutiny, but does not alone prove a bug. A broad stable neighbourhood is useful sensitivity evidence, not a substitute for evaluation on observations that did not shape the search.

Example: a +18R lookback between −3R and −2R neighbours

Three neighbouring lookbacks produce −3R, +18R and −2R under otherwise identical hypothetical assumptions. Preserve all three results. Check whether a small number of fills, a boundary convention or a data anomaly explains the peak before proposing a causal story.

The original grid has 5 × 4 × 3 = 60 configurations. Recording two predeclared cost scenarios gives 120 configuration/scenario rows. Those rows are not 120 independent tests. Selecting the cheaper cost model because it makes the winner look better is another research choice that must be disclosed.

Mistake: using a robustness check to pick the published rule

Calling a second pass a robustness check does not make it independent if its result determines which rule you publish. Predeclare the role of each scenario, and retain failed runs rather than quietly dropping them from the denominator.