ml-model-evaluation

ML Model Evaluation

Overview

Rigorous model evaluation is the foundation of credible ML research. This skill covers the complete evaluation pipeline: how to split data correctly, which metrics to report for which tasks, what baselines are required, how to structure ablation studies, and how to verify that results are statistically meaningful. Apply this skill before writing any results section or claiming model performance.

When to Use This Skill

Use this skill when:

• Selecting cross-validation strategy for a new experiment
• Choosing which metrics to compute and report for a task
• Setting up baseline comparisons before training models
• Designing ablation studies to identify component contributions
• Assessing whether train/val/test splits are clean and leak-free
• Tuning hyperparameters with nested cross-validation
• Reporting confidence intervals and checking statistical significance
• Completing the results section of a paper or technical report

---

Hard Rules (follow for every experiment)

1. NEVER report accuracy alone for imbalanced datasets — always pair with F1, AUC-PR, or MCC
2. ALWAYS compare against at least one trivial baseline before claiming model performance
3. ALWAYS use cross-validation (or a proper held-out test set) — never evaluate on training data
4. ALWAYS report confidence intervals or standard deviation across folds
5. For ablation studies: change ONE component at a time and report the delta

---

Cross-Validation Taxonomy

Choose the CV strategy based on your data structure. Wrong CV choice is one of the most common sources of inflated reported performance.

Standard K-Fold

Use when: data is IID, classes are balanced, no group structure.

from sklearn.model_selection import KFold, cross_validate
from sklearn.ensemble import RandomForestClassifier
import numpy as np

kf = KFold(n_splits=5, shuffle=True, random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42)

results = cross_validate(
    model, X, y,
    cv=kf,
    scoring=['accuracy', 'f1_macro', 'roc_auc'],
    return_train_score=True
)

print(f"Val accuracy:  {results['test_accuracy'].mean():.3f} ± {results['test_accuracy'].std():.3f}")
print(f"Train accuracy: {results['train_accuracy'].mean():.3f} ± {results['train_accuracy'].std():.3f}")
# Large train-val gap → overfitting