model-evaluation

Model Evaluation

Overview

The wrong metric on the wrong split produces confident, wrong conclusions. Evaluation is about choosing a metric that matches the business cost, validating it on a split that mirrors production, and reporting it honestly with uncertainty.

When to use

• Selecting how to score a model.
• A model "looks great" but you're unsure it's real.
• Comparing candidate models for promotion.

Metric selection

Problem	Default metric	Use when
Balanced classification	ROC-AUC, accuracy	classes ~balanced
Imbalanced classification	PR-AUC, F1, recall@k	rare positives (fraud, disease)
Probabilistic output	Log loss, Brier, calibration	you need trustworthy probabilities
Ranking	NDCG, MAP, MRR	recommendation/search
Regression	MAE (robust), RMSE (penalize big errors)	match error cost
Regression, multiplicative	MAPE / RMSLE	errors scale with magnitude

Cross-validation strategy

• Default: StratifiedKFold for classification.
• Time series: TimeSeriesSplit — never shuffle; train on past, validate on future.
• Grouped data (multiple rows per user): GroupKFold so the same group never spans train and test.
• Small data: repeated CV; report mean ± std.

Honest reporting

from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np

cv = StratifiedKFold(5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring="average_precision")
print(f"PR-AUC: {scores.mean():.3f} ± {scores.std():.3f}")  # always report spread

Beyond a single number

• Confusion matrix / classification report at the chosen threshold — accuracy hides per-class failure.
• Threshold tuning — default 0.5 is rarely optimal; pick it from the PR curve to match precision/recall needs.
• Calibration — CalibratedClassifierCV or reliability curves when probabilities feed decisions.
• Slice metrics — evaluate on subgroups to catch fairness/robustness gaps.