ensemble-model-correlation-trap

Ensemble Model Correlation Trap

Problem

In machine learning competitions, practitioners often assume that combining multiple models (stacking, weighted averaging, voting) will automatically improve performance. However, when models have extremely high correlation (>0.999), ensembling provides no benefit because the models are making nearly identical predictions.

This is a common trap in tabular data competitions where XGBoost, CatBoost, and LightGBM often converge to similar solutions.

Context / Trigger Conditions

Symptoms that indicate this problem:

• Stacking/ensemble OOF score ≤ best single model score
• Weighted average performs worse than simple average
• Logistic Regression learns nearly equal weights for all models
• High correlation (>0.995) between model predictions
• Single model continues to outperform complex ensembles

Common scenarios:

• Tabular data with tree-based models (XGBoost, CatBoost, LightGBM)
• Models trained on similar feature sets
• Competition datasets where feature space is well-explored

Solution

Step 1: Calculate Model Correlation

After training models with cross-validation:

import numpy as np
from scipy.stats import pearsonr

# Assume oof_pred1, oof_pred2, oof_pred3 are OOF predictions from different models
corr_12, p_value = pearsonr(oof_pred1, oof_pred2)
corr_13, p_value = pearsonr(oof_pred1, oof_pred3)
corr_23, p_value = pearsonr(oof_pred2, oof_pred3)

print(f"Model 1 vs 2: {corr_12:.5f}")
print(f"Model 1 vs 3: {corr_13:.5f}")
print(f"Model 2 vs 3: {corr_23:.5f}")

Step 2: Interpret Correlation Values

Correlation Range	Diagnosis	Action
< 0.99	✅ Low correlation	Ensembling likely effective
0.99 - 0.995	⚠️ Medium correlation	Ensembling may help marginally
0.995 - 0.999	⚠️ High correlation	Ensembling may not help
> 0.999	❌ Extreme correlation	Ensembling will not help

Step 3: Verify with Stacking