performance-scalability

Performance and Scalability Skill

Overview

This skill identifies, measures, and eliminates performance bottlenecks, memory leaks, and architectural scaling limits. It acts as a Pragmatic Performance Architect and Systems Engineer, focusing on maximizing throughput, minimizing latency, and achieving linear $O(N)$ or sub-linear algorithmic complexity. It balances engineering effort against empirical metrics, eliminating premature optimization while eradicating choke points under concurrent load.

When to Use

Trigger Scenarios

• Profiling slow function execution paths, API endpoints, or batch jobs.
• Diagnosing and resolving database N+1 queries, unindexed scans, and missing connection pools.
• Identifying memory leaks, unbounded cache growth, or excessive heap consumption.
• Refactoring high-complexity algorithms (e.g. converting $O(N^2)$ loops to $O(N)$ or $O(1)$ lookups).
• Designing caching strategies (Cache-Aside, Write-Through, Redis, TTL eviction).

When NOT to Use

• Premature optimization on unmeasured prototypes: Implement cleanly first using code-implementer.
• Database schema migration planning: Route to database-migration-expert.
• General code smells diagnosis without profiling: Route to code-smells-expert.
• Dedicated BigQuery query plan optimization: Route to bigquery-query-optimization.

Process

Phase 1: Automated Bottleneck Scanning

Statically scan the codebase to identify nested complexity loops, unbuffered file I/O, or N+1 query signatures:

python3 ./skills/performance-scalability/scripts/measure_performance.py scan [optional_path_to_scan]

Phase 2: Runtime Execution & Memory Profiling

Measure actual runtime execution duration and peak memory footprint of specific functions:

python3 ./skills/performance-scalability/scripts/measure_performance.py profile <path_to_script.py> <function_name>

Phase 3: Scalability Architectural Optimization

Apply proven patterns from the scalability reference:

1. Database Queries: Eliminate N+1 queries using eager loading (select_related, joinedload). Ensure query filters match composite indexes. Implement database connection pooling.
2. Caching Strategy: Apply Cache-Aside with strict TTL invalidation and LRU eviction on read-heavy paths.
3. Asynchronous Offloading: Offload long-running I/O or CPU operations into background task queues (Celery, async workers).
4. Algorithmic Complexity: Replace quadratic nested loops with hash map or set-based lookup tables.