leakage-guard

Leakage Guard

Systematically audit your feature pipeline for data leakage before any model training. Covers target leakage, temporal leakage, and cross-validation strategy alignment.

Why a dedicated skill: Leakage is the most dangerous silent failure in production ML. It inflates validation metrics by 5-40%, produces models that appear excellent in development but fail at deployment. Each type requires different detection methods.

Iron Laws

• NO FEATURE MAY USE INFORMATION FROM THE FUTURE RELATIVE TO THE PREDICTION POINT
• NO TRANSFORMER MAY BE FIT ON ANY DATA THE MODEL WILL BE ASKED TO PREDICT
• CV STRATEGY MUST MATCH DATA GENERATING PROCESS (temporal data → time-based split)

Three Types of Leakage to Check

Type	Description	Common Sources
Target Leakage	Feature derived from or correlated with target after event	Post-event flags, derived aggregates using final outcome
Temporal Leakage	Future data used to predict the past	Rolling windows that include current row, lag features with wrong offset
Preprocessing Leakage	Transformers fit on full dataset before splitting	Scalers, encoders, imputers fit before train_test_split

Step-by-Step Audit

Step 1 — Map the prediction point

Before examining any code, answer:

Prediction point: [the exact moment in time when the model makes a prediction]
Target event:     [what happens after the prediction point that defines the label]
Feature window:   [the time window of data allowed as model inputs]

Example:

Prediction point: January 1st, 2024 (start of month)
Target event:     Customer churns during January 2024
Feature window:   Data from before January 1st, 2024 only

Write this down before Step 2.

Step 2 — Audit for Target Leakage

Check every feature against the prediction point. For each feature column, answer: "Could this value change after the prediction point AND be caused by the target event?"