model-evaluation

Model Evaluation

Rigorous, statistically grounded evaluation of trained models. The test set is sacred — touched exactly once.

Iron Law: NO CONCLUSIONS WITHOUT SIGNIFICANCE TESTING. TEST SET IS EVALUATED EXACTLY ONCE.

Checklist

1. Final model refit — retrain chosen model on full training set
2. One-time test set evaluation — never to be repeated
3. Full metric suite — primary, secondary, and diagnostic metrics
4. Confidence intervals — bootstrap CIs for all reported metrics
5. Calibration check — are predicted probabilities trustworthy?
6. Error analysis — where does the model fail?
7. Feature importance — SHAP values for top features
8. Comparison to baseline — is improvement statistically significant?
9. Save evaluation artifacts — report, plots, model file
10. Flag production readiness — explicit go/no-go with criteria

Step 1: Final Model Refit

import random
import numpy as np
random.seed(42)
np.random.seed(42)

# Retrain on FULL training set (not a validation split)
final_model = LGBMClassifier(**best_params, random_state=42)
final_model.fit(X_train, y_train)

print("✅ Final model trained on full training set")
print(f"Training set size: {len(X_train)} samples")

Step 2: One-Time Test Set Evaluation

# THIS CODE BLOCK RUNS EXACTLY ONCE
y_pred = final_model.predict(X_test)
y_prob = final_model.predict_proba(X_test)[:, 1]  # for binary classification

from sklearn.metrics import (
    classification_report, confusion_matrix,
    f1_score, roc_auc_score, average_precision_score
)

print("=" * 50)
print("FINAL TEST SET EVALUATION (one-time)")
print("=" * 50)
print(classification_report(y_test, y_pred))
print(f"ROC-AUC:   {roc_auc_score(y_test, y_prob):.4f}")
print(f"PR-AUC:    {average_precision_score(y_test, y_prob):.4f}")
print(f"F1-macro:  {f1_score(y_test, y_pred, average='macro'):.4f}")
print("\nConfusion Matrix:")
print(confusion_matrix(y_test, y_pred))