ml-model-evaluation

ML Model Evaluation

Before you start

Gather the following from the user. If anything is missing, ask before proceeding:

1. What problem is the model solving? — Classification, regression, ranking, recommendation, generation
2. What is the business objective? — The real-world outcome (reduce churn, detect fraud, recommend products)
3. What data is available? — Dataset size, feature count, label quality, class balance, time range
4. What are the constraints? — Latency, model size, interpretability needs, regulatory obligations
5. What is the baseline? — Current system performance (rule-based, human, or previous model)
6. What is the cost of errors? — False positive vs false negative impact in business terms

Evaluation template

1. Define Success Metrics

Map business objectives to technical metrics. Never evaluate on technical metrics alone.

Business Objective:     Detect fraudulent transactions before settlement
Primary Metric:         Precision at 95% recall
Secondary Metrics:      AUC-ROC, F1 score, false positive rate
Business Constraint:    <50ms inference latency
Baseline Performance:   Rule-based system: 72% precision at 95% recall
Target Performance:     >85% precision at 95% recall

Metric selection rules:

• Classification: Use precision/recall/F1 for imbalanced classes. Accuracy is misleading when 98% of data is one class.
• Regression: MAE for outlier-tolerant, RMSE when large errors are disproportionately costly.
• Ranking: NDCG/MAP when order matters, precision@k when only top results matter.
• Always include a business metric: revenue impact, time saved, error cost reduction.

2. Data Splitting Strategy

Random split — Default for i.i.d. data: Train 70% / Validation 15% / Test 15%.

Temporal split — Required for time-dependent data: Train before T1 / Validation T1-T2 / Test after T2.

Stratified split — Required for imbalanced classification: maintain class proportions across splits.