data-profiling

Skill: Data Profiling

Purpose

Deep-profile the active dataset to understand schema structure, value distributions, temporal patterns, correlations, completeness gaps, and anomalies. Produces a comprehensive profile report that serves as the foundation for analysis planning and data quality assessment.

When to Use

• After connecting a new dataset (post-bootstrap, pre-analysis)
• Before the first analysis on any dataset
• When explicitly invoked by the user
• When the existing profile is stale (check last_profiled in manifest.yaml)

DISAMBIGUATION: this is the DEEP statistical profile (distributions, correlations, anomalies). For cross-table relationships/health and the first-contact "tell me about this data" overview, use data-map; for a single column's distribution, use distribution-profiler; for a plain schema listing, use /data (data-inspect).

Instructions

Step 1: Connect and Profile Schema

from helpers.data.data_helpers import get_connection_for_profiling
from helpers.data.schema_profiler import profile_source

# Get connection (auto-detects DuckDB vs CSV from active dataset)
conn_info = get_connection_for_profiling()

# Run full schema profile — introspects all tables: column names, types,
# nullability, row counts, sample values, basic statistics, date detection
schema = profile_source(conn_info)

Record the output. schema contains the full table inventory with column-level metadata. Use this to identify:

• Which tables exist and their row counts
• Which columns are date columns (for temporal analysis in Step 2)
• Which columns are numeric (for distribution and correlation analysis)
• Which columns have nulls (for completeness deep-dive in Step 2)

Step 2: Run Deep Profiling per Table

For each table in the schema, load the data and run the deep profiling functions. Prioritize tables with the most rows and the most date/numeric columns.

from helpers.data.data_helpers import read_table
from helpers.data.deep_profiler import (
    profile_distributions,
    profile_temporal_patterns,
    profile_completeness,
)

for table_info in schema["tables"]:
    table_name = table_info["name"]
    df = read_table(table_name)

    # Distribution analysis on all numeric columns
    distributions = profile_distributions(df)

    # Completeness assessment — null rates, zeros, empty strings, constant cols
    completeness = profile_completeness(df)

    # Temporal pattern analysis (only if the table has date columns)
    temporal = None
    if table_info.get("date_columns"):
        primary_date = table_info["date_columns"][0]
        temporal = profile_temporal_patterns(df, primary_date, freq="D")