python-performance-optimization

Python Performance Optimization

Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.

When to Use This Skill

• Identifying performance bottlenecks in Python applications
• Reducing application latency and response times
• Optimizing CPU-intensive operations
• Reducing memory consumption and memory leaks
• Improving database query performance
• Optimizing I/O operations
• Speeding up data processing pipelines
• Implementing high-performance algorithms
• Profiling production applications

Core Concepts

1. Profiling Types

• CPU Profiling: Identify time-consuming functions
• Memory Profiling: Track memory allocation and leaks
• Line Profiling: Profile at line-by-line granularity
• Call Graph: Visualize function call relationships

2. Performance Metrics

• Execution Time: How long operations take
• Memory Usage: Peak and average memory consumption
• CPU Utilization: Processor usage patterns
• I/O Wait: Time spent on I/O operations

3. Optimization Strategies

• Algorithmic: Better algorithms and data structures
• Implementation: More efficient code patterns
• Parallelization: Multi-threading/processing
• Caching: Avoid redundant computation
• Native Extensions: C/Rust for critical paths

Quick Start

Basic Timing

import time

def measure_time():
    """Simple timing measurement."""
    start = time.time()

    # Your code here
    result = sum(range(1000000))

    elapsed = time.time() - start
    print(f"Execution time: {elapsed:.4f} seconds")
    return result

# Better: use timeit for accurate measurements
import timeit

execution_time = timeit.timeit(
    "sum(range(1000000))",
    number=100
)
print(f"Average time: {execution_time/100:.6f} seconds")

Profiling Tools

Pattern 1: cProfile - CPU Profiling

import cProfile
import pstats
from pstats import SortKey

def slow_function():
    """Function to profile."""
    total = 0
    for i in range(1000000):
        total += i
    return total

def another_function():
    """Another function."""
    return [i**2 for i in range(100000)]

def main():
    """Main function to profile."""
    result1 = slow_function()
    result2 = another_function()
    return result1, result2

# Profile the code
if __name__ == "__main__":
    profiler = cProfile.Profile()
    profiler.enable()

    main()

    profiler.disable()

    # Print stats
    stats = pstats.Stats(profiler)
    stats.sort_stats(SortKey.CUMULATIVE)
    stats.print_stats(10)  # Top 10 functions

    # Save to file for later analysis
    stats.dump_stats("profile_output.prof")