pandas-performance

pandas - Performance & Memory Management

Standard pandas code is often memory-hungry and slow. This sub-skill provides the techniques to make pandas 10x faster and use 5x less RAM by understanding its internal architecture (BlockManager and Arrow backend).

When to Use

• Your DataFrame is larger than 1GB and causes RAM pressure.
• pd.read_csv is taking too long to load data.
• Row-wise operations (apply, iterrows) are creating bottlenecks.
• You need to perform complex joins or lookups on millions of rows.
• Preparing data for high-performance ML models.

Reference Documentation

• Official Performance Guide: https://pandas.pydata.org/docs/user_guide/enhancingperf.html
• Scaling to Large Data: https://pandas.pydata.org/docs/user_guide/scale.html
• Search patterns: df.memory_usage, pd.to_numeric(downcast=...), pd.Categorical, DataFrame.eval()

Core Principles

RAM is the Bottleneck

Pandas usually creates copies of data during operations. To handle large data, you must minimize copies and use the most efficient bit-width for your data types.

Vectorization vs. Loops

• Level 1 (Best): Built-in NumPy/Pandas vectorized functions.
• Level 2 (Good): df.eval() or df.query() for complex math.
• Level 3 (Average): np.vectorize or df.apply() (only if logic is complex).
• Level 4 (Worst): iterrows() or itertuples().

Memory Optimization Patterns

1. The "Downcasting" Workflow

Standard integer and float columns use 64 bits by default. Most scientific data fits in 16 or 32 bits.

import pandas as pd
import numpy as np

def optimize_memory(df):
    start_mem = df.memory_usage().sum() / 1024**2
    
    for col in df.columns:
        col_type = df[col].dtype
        
        if col_type != object:
            c_min = df[col].min()
            c_max = df[col].max()
            if str(col_type)[:3] == 'int':
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = df[col].astype(np.int8)
                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                    df[col] = df[col].astype(np.int16)
            else:
                if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                    df[col] = df[col].astype(np.float32)
        else:
            # Convert low-cardinality strings to Categorical
            num_unique = df[col].nunique()
            if num_unique / len(df) < 0.5:
                df[col] = df[col].astype('category')
                
    end_mem = df.memory_usage().sum() / 1024**2
    print(f'Memory reduced by {100 * (start_mem - end_mem) / start_mem:.1f}%')
    return df