auditing-deid-leakage

Auditing de-id leakage

De-identification is verified, not assumed. A model-driven redaction can miss a structured identifier (an SSN typo'd with spaces, an account number in a footer, a date in an odd format) — and a single residual identifier defeats the whole release. This skill is the adversarial second pass: scan the output of de-identification for anything that still looks like an identifier, score it, and block release on any leak. It is the verification half of OpenMed's leakage-first ethos — gate on leakage, not on F1.

When to use

• Right after deidentifying-clinical-text, before the de-identified text leaves a trust boundary (export, share, train, publish).
• When the user wants proof that "no PHI leaked," a release gate, or a CI check that fails the build if any identifier survives.
• As a belt-and-suspenders detector independent of the model that produced the redaction — a deterministic checker catches different failures than the NER.

Run this on the de-identified text, not the original. The original is expected to be full of identifiers.

Quick start

Two complementary passes — a deterministic structural scan plus a model second-pass diff:

import re
import openmed

# Synthetic — the de-identified OUTPUT we are auditing for residual leaks.
deid_text = "Patient [NAME] seen on [DATE]. Backup contact 415-555-0184; acct 4111111111111111."

def luhn_ok(digits: str) -> bool:
    nums = [int(d) for d in digits]
    nums[-2::-2] = [(2 * d - 9 if 2 * d > 9 else 2 * d) for d in nums[-2::-2]]
    return sum(nums) % 10 == 0

DETECTORS = {
    "SSN":   (r"\b\d{3}-\d{2}-\d{4}\b", "critical", None),
    "EMAIL": (r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b", "high", None),
    "PHONE": (r"\b(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b", "high", None),
    "DATE":  (r"\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b", "medium", None),
    "MRN":   (r"\bMRN[:#\s]*\d{5,}\b", "high", None),
    "CARD":  (r"\b(?:\d[ -]?){13,19}\b", "critical", luhn_ok),  # checksum-gated
}

findings = []
for label, (pattern, severity, checksum) in DETECTORS.items():
    for m in re.finditer(pattern, deid_text, flags=re.IGNORECASE):
        token = m.group()
        if checksum and not checksum(re.sub(r"\D", "", token)):
            continue  # fails Luhn -> not a real card number, skip
        findings.append({"label": label, "severity": severity,
                         "start": m.start(), "end": m.end()})  # offsets, not text

# Second-pass model detector: re-run PII extraction on the de-id output.
residual = openmed.extract_pii(deid_text)          # PredictionResult
for ent in residual.entities:
    findings.append({"label": ent.label, "severity": "high",
                     "start": ent.start, "end": ent.end})

leaked = bool(findings)
print({"leak": leaked, "count": len(findings)})    # report carries NO plaintext
assert not leaked, "Release BLOCKED: residual identifiers detected."