imbalanced-data

Imbalanced Data

Overview

When positives are rare, naive training and naive metrics both mislead. A model that always predicts "negative" can score 99% accuracy and catch zero fraud. Handle imbalance at three levels: metric, algorithm, and threshold — and resample inside cross-validation, never before.

When to use

• Class ratio is skewed (e.g., 95/5 or worse).
• Catching the rare class matters (fraud, defaults, rare disease, defects).

Step 1 — Fix the metric first

Drop accuracy and ROC-AUC-as-sole-metric. Prefer:

• PR-AUC (average precision) — most informative for rare positives.
• Recall @ fixed precision — "catch X% of fraud while keeping false alarms tolerable."
• F1 / Fβ — β>1 weights recall when misses are costly.

Step 2 — Algorithm-level handling

Technique	How	When
Class weights	class_weight="balanced" / scale_pos_weight	First choice — no data duplication
Undersample majority	RandomUnderSampler	Lots of data, majority redundant
Oversample minority	SMOTE / ADASYN	Limited minority samples
Combine	SMOTE + Tomek/ENN	Noisy boundaries

Class weights are the cheapest, leak-free first move:

# sklearn
LogisticRegression(class_weight="balanced")
# XGBoost / LightGBM
scale_pos_weight = n_negative / n_positive

Step 3 — Resample INSIDE the pipeline (no leakage)

Resampling before CV leaks synthetic neighbors across the split and inflates scores. Use imblearn's pipeline so SMOTE fits on training folds only:

from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

pipe = ImbPipeline([
    ("smote", SMOTE(random_state=42)),   # applied to train fold only
    ("clf", HistGradientBoostingClassifier(random_state=42)),
])
cv = StratifiedKFold(5, shuffle=True, random_state=42)
print(cross_val_score(pipe, X, y, cv=cv, scoring="average_precision").mean())