eda-and-data-profiling

EDA and Data Profiling

Overview

Exploration answers "what is actually in this data?" before anything is built on it. In this role exploration has a deliverable: the facts you learn become a schema, a test or a documented assumption in the repository — not a notebook cell nobody re-runs.

Core principle: every surprise you find in EDA ends up as a check that would catch it next time (data-validation-contracts) or a written assumption in your closing message.

1. Structure first

Answer these before looking at any distribution:

• Grain: one row is one what? (an order, an order line, a customer-day)
• Key: which column(s) identify a row — and are they actually unique?
• Size and span: row count, earliest and latest timestamp, any missing days.
• Freshness: when was it last updated; is the latest period complete?
• Column roles: identifier, dimension, metric, timestamp, free text, boolean, nested.

def assert_grain(df: pd.DataFrame, key: list[str]) -> None:
    dupes = df.duplicated(subset=key, keep=False)
    if dupes.any():
        raise ValueError(f"{int(dupes.sum())} rows share a {key} key, e.g. {df.loc[dupes, key].head(3).to_dict('records')}")

2. Profile every column

Column kind	Look at
All	null rate, distinct count, cardinality ratio, top and bottom values
Numeric	min, p1, p25, median, mean, p75, p99, max; zeros; negatives where impossible
String / category	empty strings, whitespace, case variants ("US", "us", "USA"), unexpected categories
Timestamp	min/max, future dates, gaps per day/week, timezone (naive vs aware)
Target	class balance or distribution, by time period

Fast paths:

duckdb.sql("SUMMARIZE SELECT * FROM 'data/events/*.parquet'").df()

def profile(df: pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame({
        "dtype": df.dtypes.astype(str),
        "null_rate": df.isna().mean(),
        "n_unique": df.nunique(dropna=True),
        "example": df.apply(lambda s: s.dropna().iloc[0] if s.notna().any() else None),
    })