data-validation

Data Validation

Schema-based and statistical validation of datasets before any downstream use.

Iron Law: NO TRAINING WITHOUT DATA QUALITY VALIDATION

Checklist

1. Define or load schema — column names, dtypes, nullable flags, value ranges
2. Run structural checks — shape, columns present, no unexpected columns
3. Run type checks — each column matches expected dtype
4. Run range checks — numeric columns within expected bounds
5. Run categorical checks — only expected categories present
6. Run null checks — nullability constraints satisfied
7. Run statistical checks — distributions within expected drift thresholds
8. Run cross-column checks — logical constraints between columns
9. Generate validation report — pass/fail per check, severity, action
10. Gate decision — BLOCK if any CRITICAL failure, WARN for others

Validation Framework

Use Pandera for Python-based schema validation:

import pandera as pa
from pandera.typing import DataFrame, Series

class CustomerSchema(pa.DataFrameModel):
    age: Series[int] = pa.Field(ge=0, le=120, nullable=False)
    income: Series[float] = pa.Field(ge=0.0, nullable=True)
    churn: Series[int] = pa.Field(isin=[0, 1], nullable=False)
    signup_date: Series[pa.DateTime] = pa.Field(nullable=False)

    class Config:
        coerce = True
        strict = True   # no extra columns allowed

# Validate
try:
    CustomerSchema.validate(df, lazy=True)
    print("✅ Validation passed")
except pa.errors.SchemaErrors as e:
    print("❌ Validation failed:")
    print(e.failure_cases)

Structural Checks

# Required columns present
expected_cols = set(schema.columns.keys())
actual_cols = set(df.columns)
missing = expected_cols - actual_cols
extra = actual_cols - expected_cols

if missing:
    print(f"❌ CRITICAL: Missing columns: {missing}")
if extra:
    print(f"⚠️ WARN: Unexpected columns: {extra}")

# Row count sanity
if len(df) == 0:
    raise ValueError("❌ CRITICAL: Dataset is empty")
if len(df) < min_expected_rows:
    print(f"⚠️ WARN: Only {len(df)} rows, expected at least {min_expected_rows}")