model-selection

Model Selection

Systematic selection of the best model through baseline comparison and rigorous hyperparameter optimization.

Iron Law: NO HYPERPARAMETER TUNING WITHOUT BASELINE COMPARISON FIRST. METRIC MUST MATCH BUSINESS OBJECTIVE.

Do NOT begin model selection if test-driven-data-science has not been run. Check the manifest:

manifest = read_manifest()
assert manifest["data_validation"].get("decision") not in (None, "BLOCKED"), \
    "BLOCKED: test-driven-data-science must PASS before model selection. Run it first."

Checklist

1. Confirm correct metric — must match task type and business objective
2. Establish dummy baseline — minimum bar any model must beat
3. Train 4 baseline models — without tuning
4. Cross-validate all baselines — stratified k-fold, log all results
5. Statistical comparison — Wilcoxon test on CV scores
6. Select top 2 candidates — for hyperparameter optimization
7. Run Bayesian HPO — Optuna, minimum 50 trials, log to MLflow
8. Final model selection — based on CV score, NOT test set
9. Document selection rationale — model choice + justification

Step 1: Choose the Right Metric

Task	Metric (Primary)	Metric (Secondary)	Never Use
Binary classification, balanced	F1	AUC-ROC	—
Binary classification, imbalanced	F1-macro or PR-AUC	Recall	Accuracy alone
Multi-class classification	F1-macro	—	Accuracy alone
Regression	RMSE	MAE, R²	R² alone
Ranking	NDCG	MAP	Accuracy
Time series	MAPE or SMAPE	RMSE	R²

# For imbalanced classification: always check the ratio first
minority_ratio = df[target].value_counts(normalize=True).min()
if minority_ratio < 0.20:
    print(f"⚠️ Imbalanced dataset ({minority_ratio:.1%} minority)")
    print("Primary metric: F1-macro or PR-AUC — NOT accuracy")
    PRIMARY_METRIC = 'f1_macro'