data-cleaning

Data cleaning — make dirty data trustworthy, and make the cleaning auditable

A clean table is typed + deduped + normalized + validated + reproducible. The deliverable here is never "I opened a notebook and fixed some rows by hand." It is a re-runnable function clean(raw) -> df plus a schema gate that fails loud when next month's file violates the contract. Reproducible means the same input always yields the same output: versions pinned, sorts deterministic, nothing random without a seed. If you can't re-run it tomorrow and get the identical result, you haven't cleaned the data — you've edited a snapshot.

Cleaning starts once you hold tabular rows and ends at a validated table/DataFrame/Parquet. Before that boundary the job is acquisition (data-scraper, structured-extraction); after it, consumption (spreadsheet-ops, analytics, business-intelligence, forecasting). Multi-GB analytical SQL is an engine choice, not a cleaning one — duckdb.

Current stack (verified 2026-06-02): pandas 3.0.x (3.0.0 shipped 2026-01-21) and pandera 0.31.1 (supports pandas ≥ 3) for in-pipeline schema validation; Polars and DuckDB when pandas runs out of RAM. Pin them: pandas==3.0.3, pandera==0.31.1.

The pipeline shape

One canonical order. Each step is positioned for a reason, not by habit.

import pandas as pd

def clean(raw_path: str) -> pd.DataFrame:
    df = read_typed(raw_path)     # 1. read with explicit dtypes — never let pandas guess
    df = normalize(df)            # 2. strings/categories/numbers/dates — collapse invisible variance
    df = dedupe(df)               # 3. AFTER normalize+type, so "1"/1 and "US "/"US" actually collapse
    df = handle_missing(df)       # 4. decide per column: drop / impute+flag / leave NA / quarantine
    df = Schema.validate(df, lazy=True)  # 5. the GATE — fail loud, surface every violation at once
    return df