leakage-safe-feature-engineering

Leakage-Safe Feature Engineering

Overview

Leakage is information in training that will not exist at prediction time. It produces a great offline score and a model that fails in production — and no test fails, because the code does exactly what it was told. The three sources: transforms fitted on rows they later score, features computed from the future, and the same entity on both sides of a split.

Core principle: split first; everything that learns from data (scalers, imputers, encoders, selectors, resamplers) is fitted inside the training fold only — in practice, inside one scikit-learn Pipeline that is cross-validated and shipped as a single artifact.

1. Fit inside the Pipeline, never before the split

# ❌ the scaler and the selector have seen the test rows
X_scaled = StandardScaler().fit_transform(X)
X_sel = SelectKBest(k=20).fit_transform(X_scaled, y)
X_train, X_test, y_train, y_test = train_test_split(X_sel, y)

# ✅ split once, then everything learned lives in the pipeline
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=seed
)
preprocess = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore", min_frequency=20), categorical_cols),
])
model = make_pipeline(preprocess, SelectKBest(k=20), LogisticRegression(max_iter=1000))
scores = cross_validate(model, X_train, y_train, cv=cv, scoring="average_precision")

cross_validate refits the whole pipeline per fold, so imputation medians, category vocabularies and selected features are learned from that fold's training part only. Use set_output(transform="pandas") when you need column names downstream.

2. Encoders and resampling

• Target encoding: scikit-learn's TargetEncoder.fit_transform uses internal cross-fitting; fit(X, y).transform(X) on the same rows does not and leaks the target. Inside a Pipeline, fit calls fit_transform, which is the safe path. Never compute per-category target means with a groupby over the full table.
• Frequency/count encoding, aggregates, PCA, clustering features: fitted transforms like any other — inside the pipeline.
• Class imbalance: prefer class_weight="balanced" or threshold tuning. Oversampling (SMOTE) goes inside an imblearn.pipeline.Pipeline only if the repository already depends on imbalanced-learn; oversampling before the split copies near-identical rows into validation.