feature-engineering

Feature Engineering

Systematic transformation of raw data into model-ready features. Every step must be reproducible, leakage-free, and validated.

Iron Law: NO FEATURES DERIVED FROM THE TARGET VARIABLE. NO TRANSFORMERS FIT ON THE FULL DATASET.

Checklist

1. Confirm EDA and validation completed — check for EDA report and validation gate decision
2. Establish train/test split — before any transformation
3. Audit leakage candidates — review flags from EDA, remove or justify each
4. Plan transformations by feature type — numeric, categorical, datetime, text
5. Implement transformations with Pipelines — scikit-learn Pipeline or similar
6. Fit transformers on training data only — persist fitted objects
7. Validate post-transform schema — run data-validation on output
8. Log feature registry — document every feature: business motivation, transformation, MI score, leakage status
9. Run feature importance check — flag unexpectedly dominant features

Train/Test Split First

from sklearn.model_selection import train_test_split

# ALWAYS split before ANY transformation
X_train, X_test, y_train, y_test = train_test_split(
    df.drop(columns=[target]),
    df[target],
    test_size=0.2,
    random_state=42,        # REQUIRED: document seed
    stratify=df[target] if is_classification else None
)

print(f"Train: {len(X_train)} rows | Test: {len(X_test)} rows")
print(f"Train target distribution:\n{y_train.value_counts(normalize=True)}")
print(f"Test target distribution:\n{y_test.value_counts(normalize=True)}")

For time-series data: Split by time, NEVER randomly.

cutoff_date = df['date'].quantile(0.8)
train = df[df['date'] < cutoff_date]
test  = df[df['date'] >= cutoff_date]