ml-sweet-spot-principle

ML Sweet Spot Principle: "过犹不及" (More is Not Always Better)

Problem

In ML optimization, there's a common intuition that "more is better":

• More features → better performance?
• More trees → better performance?
• Lower learning rate → better performance?
• Deeper trees → better performance?

Reality: Each has an optimal point beyond which performance degrades.

Context / Trigger Conditions

Use this principle when:

• OOF score keeps improving but LB score plateaus or drops
• Adding more features/features causes performance decline
• More complex model (deeper, more trees) doesn't help
• Debate between simple vs complex approaches
• Hyperparameter tuning shows diminishing returns

Key indicator: OOF-LB gap widening as complexity increases

Solution

Step 1: Systematic Boundary Testing

Don't guess—test systematically:

# Example: Feature count search
for n_features in [15, 16, 17, 18, 19, 20, 21, 22]:
    features = top_features[:n_features]
    oof_score = cross_validate(model, X[:, :n_features], y)
    lb_score = submit_and_check(features)

    results.append({
        'n_features': n_features,
        'oof': oof_score,
        'lb': lb_score
    })

Pattern to recognize:

Too few → Underfitting (both OOF and LB low)
Sweet spot → Optimal (OOF and LB both peak)
Too many → Overfitting (OOF high, LB drops)

Step 2: Detect Overfitting via OOF-LB Gap

Scenario	OOF	LB	Interpretation	Action
Healthy	High	High	Generalizes well	Keep current setup
Overfitting	High	Low/Drop	Memorizing training	Reduce complexity
Underfitting	Low	Low	Not learning enough	Add capacity/features

Real example from S6E2:

XGBoost n2603 lr0.0378:  OOF=0.95551, LB=0.95369  ✅ Sweet spot
XGBoost n3000 lr0.032:    OOF=0.95551, LB=0.95368  ❌ Slight overfit

Both have same OOF, but the simpler configuration generalizes better!