deidentify-a-dataset

De-identify a dataset

Keep the source local, name the free-text columns explicitly, and write to a different destination. Never infer columns or print source and redacted cell values.

Procedure

1. Confirm that the input is CSV, JSONL/NDJSON, or Parquet.
2. Confirm which columns contain free text. Do not scan or log values to guess.
3. Choose a policy and language. Prefer strict_no_leak when recall is the governing safety requirement.
4. Write to a new path; never overwrite the input.
5. Inspect only result.summary, which contains aggregate counts and rates.
6. Validate recall and residual leakage on representative synthetic or approved evaluation fixtures before releasing the output.

Runnable synthetic example

Install the model runtime first with python -m pip install "openmed[hf]".

import csv
from pathlib import Path

from openmed import redact_dataset

source = Path("synthetic-notes.csv")
destination = Path("synthetic-notes.redacted.csv")

with source.open("w", newline="", encoding="utf-8") as handle:
    writer = csv.DictWriter(handle, fieldnames=["record_id", "note"])
    writer.writeheader()
    writer.writerows(
        [
            {
                "record_id": "SYNTH-001",
                "note": (
                    "Taylor Example called 212-555-0198 about a "
                    "metformin refill."
                ),
            },
            {
                "record_id": "SYNTH-002",
                "note": (
                    "Send the synthetic follow-up to "
                    "[email protected]."
                ),
            },
        ]
    )

result = redact_dataset(
    source,
    text_columns=["note"],
    output_path=destination,
    policy="strict_no_leak",
    lang="en",
)

print(result.output_path)
print(result.summary.to_dict())  # Aggregate counts only; no cell contents.

Use the equivalent CLI for an existing dataset:

openmed redact-dataset notes.csv \
  --text-columns note,comment \
  --policy strict_no_leak \
  --output notes.redacted.csv