phi-deidentifier

PHI De-identifier

De-identify clinical / health text to the HIPAA Safe Harbor standard (or a Limited Data Set) using the bundled, offline scripts/deidentify.py. Every run produces two things: the redacted text and an audit manifest listing what was removed, by category, plus anything that needs human review.

When to use

• "De-identify / anonymise / scrub / redact this note (or file / folder)."
• "Remove patient identifiers before I share this."
• "Make this Safe Harbor compliant" or "produce a limited data set."
• Preparing PHI-free samples for demos, test fixtures, analytics, or model prompts.

Do not use it to answer clinical questions or to claim legal certification — it is a mechanical redactor plus an audit trail for a human to verify.

How to run it

Always call the bundled script; never improvise redaction logic.

# Safe Harbor (default): writes NOTE.deid.txt + NOTE.manifest.json next to input
python scripts/deidentify.py NOTE.txt

# Limited Data Set: retains dates and city/state/ZIP, strips direct identifiers
python scripts/deidentify.py NOTE.txt --mode limited

# Inline text, machine-readable result to stdout
python scripts/deidentify.py --text "John Doe, SSN 123-45-6789" --json

# HYBRID name recall: pass the person names YOU (the agent) spotted in free prose
python scripts/deidentify.py --text "<text>" --names "Rafael Alcaraz,Ngozi Okafor" --json

# STRUCTURED / JSON input: redact by FIELD NAME (MRN, City, Employer, DOB, ...)
# Use this whenever the input is a JSON object or a key/value record — NOT free text.
python scripts/deidentify.py intake_record.json --format record --json

# ALSO emit a reversible token->value crosswalk (SENSITIVE — re-identifying)
python scripts/deidentify.py NOTE.txt --map crosswalk.json

Steps:

1. Identify the input (a file path, a folder, or inline text) and the target standard: Safe Harbor (default) or Limited Data Set (--mode limited).
2. Pick the right mode for the shape of the input. If the input is a structured record — a JSON object, or clear Field: value key/value pairs (e.g. an intake record, an API payload, an exported row) — run it with --format record. Record mode maps each field by its NAME (MRN, City, Employer, DOB, SSN, …) and redacts deterministically; free-text mode will MISS quoted/structured identifiers like "MRN": "33108876" because there is no adjacent label to anchor on. Use the default (free-text) mode only for prose. Still run the name-recall pass on any free-text field VALUES (e.g. a ChiefComplaint or Notes field that contains a person's name).
3. Name-recall pass (do this every time on free prose). Before running, read the text yourself and list every PERSON name and PLACE (city / facility) you can identify — including unusual or non-English names that a regex or the bundled lexicon would miss (e.g. "Rafael Alcaraz", "Okafor"). This is the model/agent contribution to the hybrid; you are the recall booster.
4. Run scripts/deidentify.py, passing the names from step 3 via --names "Name One,Name Two" (or a --names-file with {"names":[...], "places":[...],"orgs":[...]}). The script deterministically redacts every occurrence, tokenizes consistently, runs its guard, and writes the manifest. The script — not your paraphrase — is the auditable system of record.
5. Return the redacted text exactly as produced, and summarise the manifest: how many identifiers of each category were removed.
6. Surface needs_human_review. Names caught heuristically, and any unknown capitalised text flagged for review, are listed there — tell the user these must be verified before release, and that bare names in free prose may still be missed.
7. Only write the --map crosswalk when the user explicitly needs to re-identify later; warn that it is sensitive and must be stored securely, separately from the de-identified output.