polars

Polars - High-Performance Dataframes

Polars is designed for speed. Unlike pandas, which processes data sequentially on a single CPU core, Polars parallelizes operations across all available cores. Its "Lazy API" allows it to optimize queries before execution, significantly reducing memory overhead and processing time.

When to Use

• Processing large datasets (1GB - 100GB+) that struggle in pandas.
• When execution speed is a priority (Polars is often 10-100x faster than pandas).
• Working with complex data transformation pipelines (Lazy evaluation).
• Systems with limited RAM (Polars is more memory-efficient than pandas).
• Situations requiring strict type safety and consistent null handling.
• Reading/writing large Parquet, CSV, or Avro files.

Reference Documentation

Official docs: https://docs.pola.rs/
User Guide: https://docs.pola.rs/user-guide/
Search patterns: pl.DataFrame, pl.LazyFrame, pl.col, df.select, df.filter, df.group_by

Core Principles

Eager vs. Lazy API

• Eager: Operations are executed immediately (like pandas).
• Lazy: Operations are queued into a query plan. Polars optimizes the plan (e.g., predicate pushdown, projection pushdown) and executes it only when called.

The Expression API

Polars uses a declarative syntax. Instead of writing loops or complex lambdas, you write expressions using pl.col(). These expressions are highly optimized and run in parallel.

Apache Arrow

Polars stores data in the Apache Arrow format, enabling zero-copy data exchange with other tools like PyArrow and DuckDB.

Quick Reference

Installation

pip install polars
# For Excel/Cloud support
pip install 'polars[all]'

Standard Imports

import polars as pl
import numpy as np

Basic Pattern - Lazy Workflow (The "Polars Way")

import polars as pl

# 1. Scan (Lazy) - doesn't load data yet
lf = pl.scan_csv("massive_data.csv")

# 2. Build Query Plan
query = (
    lf.filter(pl.col("age") > 25)
    .group_by("city")
    .agg([
        pl.col("salary").mean().alias("avg_salary"),
        pl.col("name").count().alias("count")
    ])
    .sort("avg_salary", descending=True)
)

# 3. Collect (Execute)
df = query.collect()