model-assessment

Model-Assessment Skill

Assess a trained medical-imaging model (in-house, vendor or open-weights; segmentation, classification or detection) in the order the evidence is built: Part A designs and audits the validation study, Part B computes task-correct held-out metrics, Part C adds the uncertainty / OOD / abstention layer a deployment claim needs, and Part D makes an explainability analysis survive review. Run the parts the request needs (a Grad-CAM question starts at Part D), but no metric headline is reported before the Part A split gate is green. Each part ends in a stdlib gate whose verdict is reproduced from a file — never report a pass without running it.

Numbers come only from code executed on the supplied predictions, split table, or the researcher's executed UQ/XAI code; if predictions or ground truth are missing, say so and stop. Integrate MONAI / nnU-Net, MAPIE, captum, pytorch-grad-cam and pretrained OOD scorers by reference — never reimplement them, never build or train the model, never run a model on real patient data.

Elsewhere: building/training → /model-scaffold; choosing or vetting the model → /model-selection; data-stage preprocessing leakage → /imaging-data; paired model comparison / added value over a baseline / decision curves / MRMC / ICC / calibration tables → /analyze-stats (added value: incremental_value.md); AI-vs-expert reader rubric and IRR → /design-ai-benchmarking; LLM/MLLM → /mllm-eval; general validity → /design-study; classical-ML tabular calibration → /radiomics-ml; item-by-item reporting audit → /check-reporting; a finished manuscript → /self-review or /peer-review (the MD0–MD11 model_development.md probe).

Part A — Validation design

The rationale behind Phases 2–7 — the full leakage taxonomy, the internal-vs-external ladder, comparator design, run variance, test-set sizing, the reporting map — is in ${CLAUDE_SKILL_DIR}/references/validation_design.md (load on demand). Verify citations via /search-lit (confirmed DOI/PMID), else mark [UNVERIFIED - NEEDS MANUAL CHECK]; flag an uncertain CLAIM 2024 / TRIPOD+AI / Metrics Reloaded item [VERIFY] and ask.