segmenting-clinical-sections

Segmenting clinical sections

A clinical note is not flat text — it is a sequence of named sections (Chief Complaint, HPI, Past Medical History, Medications, Allergies, Assessment & Plan). The same phrase means different things in different sections: "diabetes" in PMH is historical context, "diabetes" in Assessment & Plan is an active problem, and "penicillin" under Allergies is an adverse-reaction flag, not a current medication. Splitting the note into canonical sections before NER or de-identification gives every downstream OpenMed step the context it needs to be more precise — and lets you process sensitive sections under stricter policies.

When to use

• You have a free-text note, H&P, progress note, or discharge summary and are about to run NER (extracting-clinical-entities) or de-identification.
• The user wants section detection, header parsing, LOINC section mapping, or per-section processing (e.g. "redact the Social History section harder").
• Downstream NER is over- or under-firing because it can't tell historical PMH mentions from active A&P problems.

Quick start

import re
import openmed

# Synthetic note.
note = """CHIEF COMPLAINT: chest pain.
HPI: 54M with 2 hours of substernal pressure.
PAST MEDICAL HISTORY: type 2 diabetes, prior MI 2019.
MEDICATIONS: metformin 500 mg BID.
ALLERGIES: penicillin (rash).
ASSESSMENT AND PLAN: acute coronary syndrome; start aspirin, admit."""

# Map common header variants -> canonical section + LOINC document-section code.
SECTION_MAP = {
    "chief complaint": ("Chief Complaint", "10154-3"),
    "hpi": ("History of Present Illness", "10164-2"),
    "history of present illness": ("History of Present Illness", "10164-2"),
    "past medical history": ("Past Medical History", "11348-0"),
    "medications": ("Medications", "10160-0"),
    "allergies": ("Allergies", "48765-2"),
    "assessment and plan": ("Assessment and Plan", "51847-2"),
}

HEADER_RE = re.compile(r"^(?P<h>[A-Z][A-Za-z /&]+):", re.MULTILINE)

# Split note into (canonical_label, loinc, body) chunks at each header.
chunks, matches = [], list(HEADER_RE.finditer(note))
for i, m in enumerate(matches):
    raw = m.group("h").strip().lower()
    label, loinc = SECTION_MAP.get(raw, (m.group("h").strip(), None))
    body_start = m.end()
    body_end = matches[i + 1].start() if i + 1 < len(matches) else len(note)
    chunks.append({"section": label, "loinc": loinc,
                   "text": note[body_start:body_end].strip()})

# Run NER per section — pass the section label downstream as context.
for c in chunks:
    ents = openmed.analyze_text(c["text"], model_name="disease_detection_superclinical",
                                output_format="dict")
    c["entities"] = ents