memory-optimization

Memory Optimization Skill

Transform Python code to minimize memory usage while maintaining functionality.

Workflow

1. Profile to identify memory bottlenecks (largest allocations, leak patterns)
2. Analyze data structures and object lifecycles
3. Select optimization strategies based on access patterns
4. Transform code with memory-efficient alternatives
5. Verify memory reduction without correctness loss

Memory Optimization Decision Tree

What's consuming memory?

Large collections:
├── List of objects → __slots__, namedtuple, or dataclass(slots=True)
├── List built all at once → Generator/iterator pattern
├── Storing strings → String interning, categorical encoding
└── Numeric data → NumPy arrays instead of lists

Data processing:
├── Loading full file → Chunked reading, memory-mapped files
├── Intermediate copies → In-place operations, views
├── Keeping processed data → Process-and-discard pattern
└── DataFrame operations → Downcast dtypes, sparse arrays

Object lifecycle:
├── Objects never freed → Check circular refs, use weakref
├── Cache growing unbounded → LRU cache with maxsize
├── Global accumulation → Explicit cleanup, context managers
└── Large temporary objects → Delete explicitly, gc.collect()

Transformation Patterns

Pattern 1: Class to slots

Reduces per-instance memory by 40-60%:

Before:

class Point:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

After:

class Point:
    __slots__ = ('x', 'y', 'z')

    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

Pattern 2: List to Generator

Avoid materializing entire sequences:

Before:

def get_all_records(files):
    records = []
    for f in files:
        records.extend(parse_file(f))
    return records

all_data = get_all_records(files)
for record in all_data:
    process(record)

After:

def get_all_records(files):
    for f in files:
        yield from parse_file(f)

for record in get_all_records(files):
    process(record)