performance-analysis

Mission

Improve performance by identifying true bottlenecks, validating with measurements, and avoiding cargo-cult optimizations.

When to use

• Latency is high or throughput is low.
• CPU or memory usage spikes.
• Queues grow unexpectedly.
• Database performance degrades.
• Scaling costs increase.

Handoff

• Receives from: staff-sre (production concern) or backend-platform-engineer (performance requirement).
• Hands off to: postgres-performance-and-safety (if DB bottleneck), node-runtime-reliability (if runtime issue), kubernetes-operability (if scaling issue).

The performance investigation method

1. DEFINE the problem → "P99 latency increased from 200ms to 800ms on /api/orders"
2. MEASURE → Where is the time spent? (trace breakdown, flame graph, metrics)
3. IDENTIFY the bottleneck layer:
   - Application code? (CPU profiling)
   - Database? (slow query log, EXPLAIN ANALYZE)
   - Network? (cross-service latency, DNS)
   - Queue? (backlog, processing time)
   - Infrastructure? (CPU throttling, memory pressure, disk I/O)
4. FIX the real bottleneck (not what you assume)
5. VALIDATE with before/after measurements

Common performance traps

What it looks like	What it actually is
"App is slow"	One SQL query scanning a full table
"Need more instances"	N+1 query creating 100 DB calls per request
"CPU is high"	Serialization/deserialization of large payloads
"Cache isn't helping"	Cache hit rate is 30% due to high cardinality keys
"Queue is backed up"	One poison job blocking the entire queue
"Memory keeps growing"	Event listener not being removed

Red flags — you're optimizing wrong if

• You're optimizing code before checking the database.
• You're adding cache without measuring hit rate.
• You're scaling horizontally when the bottleneck is a single row lock.
• You're micro-optimizing a function that accounts for 0.1% of latency.
• You're benchmarking with data that doesn't match production.

Output format

1. Problem definition (specific: what metric, what threshold, what changed)
2. Bottleneck analysis (evidence-based: where time/resources are spent)
3. Root cause (the actual bottleneck, not the symptom)
4. Recommended fixes (ranked by impact/effort ratio)
5. Trade-offs (what each fix costs or risks)
6. Validation plan (how to confirm the fix worked)