data-preprocessing

Data Preprocessing

Build data cleaning and preparation pipelines using Pandas, NumPy, and Pandera.

When to Use

• User needs to clean a raw CSV/Parquet/JSON dataset
• User asks for exploratory data analysis (EDA)
• User needs to handle missing values, duplicates, or type conversions
• User wants to validate data against a schema
• User needs to prepare data before feature engineering or model training

Available Operations

1. Run exploratory data analysis (EDA) on a dataset
2. Build a cleaning pipeline (dedup, nulls, types, normalization)
3. Validate data with Pandera schemas
4. Profile data quality and generate reports
5. Export cleaned data to Parquet/CSV

Multi-Step Workflow

Step 1: Install Dependencies

pip install pandas numpy pandera pyarrow openpyxl

Step 2: Load and Inspect Data

import pandas as pd
import numpy as np

# Load data (adjust path/format as needed)
df = pd.read_csv("data/raw/dataset.csv")
# or: df = pd.read_parquet("data/raw/dataset.parquet")
# or: df = pd.read_json("data/raw/dataset.json")

# Quick inspection
print(f"Shape: {df.shape}")
print(f"Columns: {list(df.columns)}")
print(df.dtypes)
print(df.head())

Step 3: Run Exploratory Data Analysis

def eda_report(df: pd.DataFrame) -> dict:
    return {
        "shape": df.shape,
        "dtypes": df.dtypes.to_dict(),
        "nulls": df.isnull().sum().to_dict(),
        "null_pct": (df.isnull().sum() / len(df) * 100).round(2).to_dict(),
        "duplicates": df.duplicated().sum(),
        "numeric_stats": df.describe().to_dict(),
        "categorical_counts": {
            col: df[col].value_counts().head(10).to_dict()
            for col in df.select_dtypes(include='object').columns
        }
    }

report = eda_report(df)
for key, value in report.items():
    print(f"\n--- {key} ---")
    print(value)

Step 4: Build and Run Cleaning Pipeline

def clean_pipeline(df: pd.DataFrame) -> pd.DataFrame:
    df = df.copy()

    # 1. Remove duplicates
    before = len(df)
    df = df.drop_duplicates()
    print(f"Removed {before - len(df)} duplicate rows")

    # 2. Fix date columns
    date_cols = [c for c in df.columns if 'date' in c.lower() or c.endswith('_at')]
    for col in date_cols:
        df[col] = pd.to_datetime(df[col], errors='coerce')

    # 3. Handle numeric nulls
    for col in df.select_dtypes(include=[np.number]).columns:
        null_pct = df[col].isnull().sum() / len(df)
        if null_pct < 0.05:
            df[col] = df[col].fillna(df[col].median())
        elif null_pct > 0.5:
            print(f"Dropping column '{col}' ({null_pct:.0%} nulls)")
            df = df.drop(columns=[col])

    # 4. Handle categorical nulls
    for col in df.select_dtypes(include='object').columns:
        df[col] = df[col].fillna('unknown')

    # 5. Normalize strings
    for col in df.select_dtypes(include='object').columns:
        df[col] = df[col].str.strip().str.lower()

    return df

df_clean = clean_pipeline(df)