benchmarking-clinical-ner

Benchmarking Clinical NER

This skill produces an honest entity-level scorecard for an OpenMed NER model: precision / recall / F1 plus a per-label error breakdown. It scores spans, not tokens, because clinical entities are multi-token ("type 2 diabetes mellitus") and token-level accuracy hides boundary errors. Reported numbers are entity-level in the seqeval tradition (CoNLL-2000 / SemEval-2013 families).

When to use this skill

• You have a gold-annotated clinical corpus and an OpenMed NER model to score.
• You want strict (exact-boundary) and partial (relaxed-overlap) span F1.
• You need per-label numbers, not one aggregate — DRUG recall ≠ DISEASE recall.
• You need to explain the errors: what was missed, what was spurious, what was mislabeled.

For PHI de-id specifically, gate on leakage with evaluating-with-leakage-gates instead of (or in addition to) F1.

Match modes

Mode	Counts a hit when…	Use for
Strict / exact	predicted span boundaries and label match gold exactly	release scoring, boundary-sensitive tasks
Partial / relaxed	predicted span overlaps gold with the right label	recall-oriented triage, tokenizer-mismatch tolerance

OpenMed exposes both: compute_exact_span_f1 (strict) and compute_relaxed_span_f1 (partial), with the full bundle in compute_metrics_bundle.

Quick start

Run a model over a user-supplied gold fixtures file and print a scorecard:

from openmed.eval import run_suite, error_report

# Fixtures: JSON list of {"id", "text", "gold_spans": [{start, end, label}, ...]}
report = run_suite(
    "eval/gold/clinical_ner.json",        # YOUR gold corpus, not bundled
    suite="golden",
    model_name="OpenMed/Disease-Detection",
    device="cpu",
)

m = report.metrics
print("exact F1 :", m["exact_span_f1"]["f1"])      # strict
print("relaxed F1:", m["relaxed_span_f1"]["f1"])    # partial
print("recall by label:", m["recall_slices"]["by_label"])

# Per-label confusion matrix + capped, no-PHI error examples.
errors = error_report(
    "OpenMed/Disease-Detection",
    "eval/gold/clinical_ner.json",
    suite_name="clinical_ner",
    example_cap=5,
)
print(errors.to_markdown())                 # confusion matrix + FN/FP tables
errors.write_json("eval/out/error_analysis.json")