overfit-check

Method

Step 1 — Confirm the baseline

The strategy_id, the baseline run_id (or run one on the default window), and its window. State the baseline Sharpe / return / max DD before probing.

Step 2 — Agree the out-of-sample windows

The probe takes several runs — one backtest of the plan's weekly allowance each, plus wall-clock time. Before starting, say how many runs it needs and which windows, and start once the user agrees; they may drop or pick windows. Windows the user named are used as given. When the user asks for a default split, this one is relative to the baseline window:

• W1 in-sample — its first half;
• W2 out-of-sample — its second half;
• W3 held-out tail — the most recent 3 months;
• W4 / W5 regimes — one high-vol stretch, one low-vol / chop stretch.

Every window must cover the strategy's warm-up. For each agreed window: keel_backtest_run(strategy_id=<id>, start_date=<start>, end_date=<end>); if still active, keel_backtest_watch(backtest_id=<run_id>). When all are terminal, keel_backtest_compare once with every window's id — the table is the comparison view.

Step 3 — Parameter perturbation (only when the user asks for depth)

Pick the 1–2 parameters that matter (a lookback, a threshold, a leverage cap — not a seed). Perturb ±20% and ±50% — four runs per parameter, stated before starting — and compare once. A Sharpe that collapses under ±20% points to a fitted parameter rather than an edge.

Step 4 — Read stability

• Sharpe spread across windows: max − min > 2.0 with any window negative is fragile. Real edges degrade smoothly.
• Sign of returns: one negative disjoint window is a yellow flag; two are red.
• Parameter sensitivity: robust to ±20% or it is not an edge.

Step 5 — Verdict

• ROBUST — stable Sharpe, positive across regimes, parameters not razor-thin.
• MIXED — some windows weak, thesis holds in most; a candidate for further out-of-sample testing.
• OVERFIT — variance too high, negative OOS, or razor-thin parameters.