feature-engineering-saturation-detection

Feature Engineering Saturation Detection

Context

Feature engineering always saturates. Every project has a "ceiling" past which new features stop helping. Continuing to optimize past saturation wastes weeks of engineering time. In a recent retail SKU recommendation project, 13 consecutive failed experiments (v31-v43) occurred before recognizing saturation at 76.5% F1 (theoretical ceiling 90.7%). This skill teaches how to detect saturation early and switch paradigms.

The core lesson: optimization without a ceiling is wasted effort. If you don't know your theoretical upper bound, you can't know if you're done.

Guidance

The 4 Saturation Signals

Signal 1: Consecutive F1 Stagnation
  - 3-4+ experiments without improvement
  - Each new feature adds ≤0.1pp
  - Action: Stop adding features, switch paradigm

Signal 2: High Correlation with Existing Features
  - New feature has Spearman correlation >0.7 with existing ones
  - Captures no new information
  - Action: Skip the feature, document why

Signal 3: Distance to Theoretical Upper Bound < 15pp
  - F1-EM (oracle ceiling) - actual F1 < 15pp
  - Historical coverage ceiling < 10pp above actual
  - Action: Optimization ROI is low, consider external data / new paradigm

Signal 4: Improvements Only From Threshold/Window Tuning
  - Last 3+ improvements came from "tweak N", "expand window", "adjust threshold"
  - No new information captured
  - Action: Architectural change needed (model class, data source, real-time signals)

Diagnostic Script

def detect_saturation(experiment_log):
    """Returns saturation status + recommended action"""

    recent = experiment_log.tail(5)

    # Signal 1: Stagnation (3+ experiments without >0.2pp improvement)
    improvements = recent['f1_diff'].tolist()
    stagnant = len([x for x in improvements[-3:] if x > 0.002]) == 0

    # Signal 3: Distance to ceiling
    upper_bound = compute_f1_em_upper_bound()  # F1-EM = oracle ceiling
    distance_to_ceiling = upper_bound - recent['f1'].iloc[-1]

    # Signal 2: Feature correlation
    new_feature_max_corr = check_feature_correlation(recent['new_features'])
    high_corr = new_feature_max_corr > 0.7

    if stagnant and distance_to_ceiling < 0.15:
        return "SATURATED — switch paradigm (external data, real-time signals, new model class)"
    elif high_corr:
        return "FEATURE REDUNDANT — try different angle or skip"
    elif distance_to_ceiling < 0.05:
        return "NEAR CEILING — declare success and ship"
    else:
        return "ACTIVE — keep optimizing features"