model-evaluation

Model Evaluation SOP

Evaluate ML models systematically: choose the right metrics, run benchmarks, analyze errors, and produce reproducible evaluation reports.

When to Use

• User wants to evaluate a model's performance on a dataset
• User wants to compare two or more models objectively
• User wants to understand where a model fails (error analysis)
• User wants to check for bias across data slices
• User wants a structured evaluation report

---

Part 1 — Metric Selection by Task Type

Classification

from sklearn.metrics import (
    classification_report, confusion_matrix,
    roc_auc_score, average_precision_score,
    f1_score, precision_score, recall_score, accuracy_score,
)
import numpy as np

def evaluate_classifier(y_true, y_pred, y_prob=None, labels=None):
    """Full classification evaluation suite."""
    print("=== Classification Report ===")
    print(classification_report(y_true, y_pred, target_names=labels))

    print("=== Confusion Matrix ===")
    cm = confusion_matrix(y_true, y_pred)
    print(cm)

    if y_prob is not None:
        # Binary
        if y_prob.ndim == 1 or y_prob.shape[1] == 2:
            prob = y_prob if y_prob.ndim == 1 else y_prob[:, 1]
            print(f"\nROC-AUC:          {roc_auc_score(y_true, prob):.4f}")
            print(f"Avg Precision:    {average_precision_score(y_true, prob):.4f}")
        else:
            # Multiclass OvR
            print(f"\nROC-AUC (macro):  {roc_auc_score(y_true, y_prob, multi_class='ovr', average='macro'):.4f}")

    print(f"\nAccuracy:         {accuracy_score(y_true, y_pred):.4f}")
    print(f"F1 (macro):       {f1_score(y_true, y_pred, average='macro'):.4f}")
    print(f"F1 (weighted):    {f1_score(y_true, y_pred, average='weighted'):.4f}")

Metric guidance:

• Balanced dataset: Accuracy is acceptable; F1 macro provides class-level fairness view.
• Imbalanced dataset: Use F1-weighted, ROC-AUC, or PR-AUC (Average Precision). Accuracy misleads.
• High recall priority (medical, fraud detection): Maximize recall; accept lower precision.
• High precision priority (spam filter, legal): Maximize precision.