pandas-polars-eda

Pandas & Polars EDA Skill Guide

This skill provides production standards, high-performance code patterns, memory optimizations, and data hygiene rules for performing Exploratory Data Analysis (EDA) using Pandas and Polars.

1. Engine Comparison: Pandas vs Polars

+-----------------------+---------------------------------------+---------------------------------------+
| Feature               | Pandas (2.0+ with PyArrow)             | Polars                                |
+-----------------------+---------------------------------------+---------------------------------------+
| **Execution Engine**  | Single-threaded eager execution       | Multi-threaded query optimization     |
| **Memory Model**      | In-memory numpy / arrow backend       | Apache Arrow columnar format native   |
| **Evaluation Mode**   | Eager only                            | Eager & Lazy evaluation (`lazy()`)    |
| **Performance**       | Moderate on datasets > 1GB            | Extremely fast (10x-30x speedups)     |
+-----------------------+---------------------------------------+---------------------------------------+

2. Automated Data Health & Missing Value Profiling

A. Polars Health Check Pipeline

import polars as pl

def profile_polars_dataframe(df: pl.DataFrame) -> pl.DataFrame:
    """Generate comprehensive dataset health report in Polars."""
    null_counts = df.null_count()
    dtypes = pl.DataFrame({"column": df.columns, "dtype": [str(d) for d in df.dtypes]})
    
    stats = df.describe()
    
    summary = dtypes.with_columns(
        null_count=pl.Series([df[col].null_count() for col in df.columns]),
        null_percentage=pl.Series([round((df[col].null_count() / df.height) * 100, 2) for col in df.columns]),
        n_unique=pl.Series([df[col].n_unique() for col in df.columns]),
    )
    return summary

# Usage
df = pl.read_parquet("sales_data.parquet")
health_report = profile_polars_dataframe(df)
print(health_report)

B. Pandas PyArrow Data Profiling