model-evaluation

Model-Evaluation Skill

Purpose

This skill makes a medical-imaging model's held-out evaluation task-correct and honest: the right metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It emits a per-case metric table that the publication statistics build on, and gates the metric choice against Metrics Reloaded (Maier-Hein & Reinke et al., Nat Methods 2024) and CLAIM 2024.

It sits between /model-validation (which audits the split / design) and /analyze-stats (which owns the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC, ECE of a softmax head); /analyze-stats owns DeLong / NRI / IDI / decision curves / MRMC. Like /analyze-stats, it generates and executes code on your predictions — numbers are never hand-typed.

When to use

• You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and subgroup slices, plus a per-case table for the manuscript statistics.

When NOT to use

• Auditing the validation design / leakage → /model-validation.
• DeLong / NRI / IDI / decision curves / MRMC reader study → /analyze-stats.
• Building / training the model → /model-scaffold; LLM / MLLM → /mllm-eval.
• Figure rendering → /make-figures.

Workflow

Phase 1 — Fix the analysis unit and the task

State the task (segmentation / classification / detection / interactive / generative) and the analysis unit the metric must respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as per-patient.

Phase 2 — Compute task-correct metrics

Generate evaluation code that computes, on the held-out predictions:

• segmentation: Dice/IoU and a boundary metric (HD95 / NSD), per structure not only a global mean, with bootstrap 95% CIs.
• classification: AUROC and AUPRC with bootstrap CIs, sensitivity/specificity, and PPV/NPV at the deployment prevalence (not a balanced set).
• detection: FROC / mAP with the IoU match criterion stated.
• interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive): the segmentation metrics above plus the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time (see the metric guide; the study design is in /design-study + /model-validation).
• generative / synthesis (image generation or modification): full-reference similarity (MSE/RMSE/PSNR/SSIM) or no-reference quality (SNR/CNR, standardized visual scores), plus a downstream-task evaluation — image quality is not clinical utility (Park et al., Radiol Med 2024). For multiclass classification, state the aggregation scheme (one-vs-rest / macro / micro / pairwise / Obuchowski); time-to-event discrimination (Harrell's C, time-dependent ROC) is handed to /analyze-stats. Add calibration (reliability diagram / ECE) and subgroup slices (the Model Card Factors). See ${CLAUDE_SKILL_DIR}/references/metric_guide.md. Emit a per-case CSV for /analyze-stats.