data-profiling

Skill: Data Profiling

Purpose

Deep-profile the active dataset to understand schema structure, value distributions, temporal patterns, correlations, completeness gaps, and anomalies. Produces a comprehensive profile report that serves as the foundation for analysis planning and data quality assessment.

Bundled profiling library: the profiling functions used below ship inside this skill at scripts/ (data_helpers.py, schema_profiler.py, deep_profiler.py; pandas/numpy, plus duckdb for DuckDB sources). Add this skill's scripts/ directory to sys.path before the imports in the steps below:

import sys
import pandas as pd
sys.path.insert(0, "<path to this skill>/scripts")  # the scripts/ dir next to this SKILL.md

If the skill install path cannot be resolved (some sandboxed environments): read the script file(s) from this skill, write a copy into a scripts/ folder inside the working folder, and run from there. The scripts are self-contained.

Run Python from the working folder root so .knowledge/ paths resolve.

When to Use

• After connecting a new dataset (post-bootstrap, pre-analysis)
• Before the first analysis on any dataset
• When explicitly invoked by the user
• When the existing profile is stale (check last_profiled in manifest.yaml)

DISAMBIGUATION: this is the DEEP statistical profile (distributions, correlations, anomalies). For cross-table relationships/health and the first-contact "tell me about this data" overview, use data-map; for a single column's distribution, use distribution-profiler; for a plain schema listing, use /data (data-inspect).

Instructions

Step 1: Connect and Profile Schema

from data_helpers import get_connection_for_profiling
from schema_profiler import profile_source

# Get connection (auto-detects DuckDB vs CSV from active dataset)
conn_info = get_connection_for_profiling()

# Run full schema profile — introspects all tables: column names, types,
# nullability, row counts, sample values, basic statistics, date detection
schema = profile_source(conn_info)