authoring-model-cards

Authoring Model Cards

A model card is the honest spec sheet for a model: what it's for, how well it works, where it breaks, and who it might fail. For clinical models this is governance-critical — an undocumented de-id model is one nobody can sign off on. This skill fills a model card directly from OpenMed eval outputs so the numbers are reproducible, not aspirational.

When to use this skill

• You're publishing or updating an OpenMed model and need its card.
• You have eval artifacts (GateReport, fairness_report, error_report) and need to turn them into intended-use, metrics, and limitations sections.
• A clinical AI governance / model-risk review needs a transparency document.

Run the evals first (see evaluating-with-leakage-gates, benchmarking-clinical-ner, auditing-subgroup-fairness); this skill documents their results — it does not generate the numbers.

Card sections (Mitchell et al., + clinical extensions)

See references/model-card-sections.md for the full section-to-source map. The load-bearing sections for an OpenMed model:

• Model details — repo id, family, tier, format, params, milestone, license (Apache-2.0). Pull from the GateReport identity fields.
• Intended use — the clinical task and the deployment envelope.
• Out-of-scope / misuse — explicitly: not a medical device; not for autonomous clinical decisions; de-id is verified, not assumed.
• Metrics — entity-level P/R/F1 and, for de-id, residual leakage + per-label recall floors and the gate decision.
• Quantitative analysis (subgroups) — per-group leakage/recall from fairness_report, including which groups lack data.
• Limitations — error patterns from error_report; calibration assumptions.
• Caveats & disclaimer — the medical-device disclaimer.

Quick start — fill the card from eval outputs

from openmed.eval import (
    run_suite, ReleaseGate, fairness_report, error_report,
)

report = run_suite("eval/gold/test.json", suite="golden",
                   model_name="OpenMed/Privacy-PII-Detection", device="cpu",
                   metadata={"family": "PII", "tier": "base",
                             "policy": "hipaa_safe_harbor"})

gate = ReleaseGate(milestone="v1.6", policy="hipaa_safe_harbor").evaluate(report)
fair = fairness_report("OpenMed/Privacy-PII-Detection", "golden")
errs = error_report("OpenMed/Privacy-PII-Detection", "eval/gold/test.json")

card = {
    "model_details": {
        "repo_id": gate.repo_id, "family": gate.family, "tier": gate.tier,
        "format": gate.format, "license": "Apache-2.0",
    },
    "metrics": {
        "exact_span_f1": report.metrics["exact_span_f1"]["f1"],
        "residual_leakage_rate": gate.residual_leakage_rate,
        "critical_leakage_count": gate.critical_leakage_count,
        "per_label_recall": dict(gate.per_label_recall),
        "release_decision": gate.decision,            # RELEASABLE / QUARANTINED
    },
    "subgroup_analysis": fair.to_dict(),              # per-group leakage/recall
    "limitations": errs.to_dict()["confusion_matrix"],
}
# Render `card` into Markdown front matter + body (or the HF card template).