model-ensemble-negative-weight-effect

Model Ensemble Negative Weight Effect

Problem

Ensembling a strong model with a weak model can actually decrease performance, even when using theoretically sound weighting strategies like exp(OOF AUC).

Context / Trigger Conditions

• Two models with significantly different validation scores (e.g., ΔAUC > 0.03)
• Weighted ensemble produces lower score than the best single model
• Weights calculated as exp(score) / sum(exp(scores))
• Temptation to ensemble "just in case" it helps
• Models using different feature sets or strategies

Solution

Rule of Thumb: Only Ensemble Similar-Performing Models

Don't ensemble when:

Model A OOF: 0.955
Model B OOF: 0.906
ΔOOF: 0.049  ← Too large! Model B will drag down Model A

Do ensemble when:

Model A OOF: 0.955
Model B OOF: 0.953
ΔOOF: 0.002  ← Similar enough, ensemble may help

Verification Method

Before committing to ensemble, test:

# Calculate theoretical weight
import numpy as np
w_a = np.exp(0.955) / (np.exp(0.955) + np.exp(0.906))  # 0.73
w_b = np.exp(0.906) / (np.exp(0.955) + np.exp(0.906))  # 0.27

# If weaker model gets >20% weight, it's risky
print(f"Weaker model weight: {w_b:.2%}")  # If >20%, be careful

Alternative Strategies

1. Skip ensemble: Use best single model
2. Threshold ensemble: Only include models within ΔX of best

   threshold = 0.01  # Only models within 1% AUC
   candidates = [m for m in models if best_score - m.score < threshold]
   

3. Blend only top-N: Take top 3 models, ignore rest
4. Stacking: Use meta-learner instead of simple weighted average

Example

S6E2 Heart Disease Competition:

Model	OOF AUC	LB AUC	Weight (exp)
V3 Single	0.95545	0.95358	-
V4.1 Raw	0.90639	~0.90	48.8%
V3+V4 Ensemble	-	0.95348	-

Result: Ensemble (0.95348) < V3 Single (0.95358) < V3 Multiseed (0.95359)

Why it failed:

• exp(0.95545) / (exp(0.95545) + exp(0.90639)) ≈ 0.512
• V4.1 got 48.8% weight despite being Δ0.049 worse!
• exp归一化在小差异下几乎等于线性归一化
• Weaker model's nearly 50% weight dragged down performance
• No complementary signal between feature engineering and raw features