python-performance-optimization

Python Performance Optimization

Profile, analyze, and optimize Python code for better performance - CPU profiling, memory optimization, and implementation best practices.

When to Invoke

• User reports slow Python code or asks to speed up execution
• Profiling or benchmarking Python applications
• Reducing CPU time, memory consumption, or I/O wait
• Optimizing database queries or data processing pipelines
• Debugging memory leaks or excessive memory usage
• Choosing between parallelization strategies (threading, multiprocessing, async)
• Evaluating algorithmic vs implementation-level improvements

Core Concepts

Profiling Types

• CPU Profiling: Identify time-consuming functions (cProfile, py-spy)
• Memory Profiling: Track memory allocation and leaks (tracemalloc, memory_profiler)
• Line Profiling: Profile at line-by-line granularity (line_profiler)
• Call Graph: Visualize function call relationships

Performance Metrics

• Execution Time: How long operations take
• Memory Usage: Peak and average memory consumption
• CPU Utilization: Processor usage patterns
• I/O Wait: Time spent on I/O operations

Optimization Strategies

• Algorithmic: Better algorithms and data structures
• Implementation: More efficient code patterns
• Parallelization: Multi-threading/processing
• Caching: Avoid redundant computation
• Native Extensions: C/Rust for critical paths

Quick Start

import time
import timeit

# Simple timing
start = time.time()
result = sum(range(1000000))
print(f"Execution time: {time.time() - start:.4f} seconds")

# Accurate benchmarking with timeit
execution_time = timeit.timeit("sum(range(1000000))", number=100)
print(f"Average time: {execution_time/100:.6f} seconds")

Profiling Tools Summary

cProfile - CPU Profiling

python -m cProfile -o output.prof script.py
python -m pstats output.prof

line_profiler - Line-by-Line

uv add --dev line-profiler    # or: uv tool install line-profiler
kernprof -l -v script.py