performance-profiling

Profile before you optimise

Performance work without a measurement is refactoring with a story attached. Reproduce the slowness, profile it, fix what the profile points at, and show the same measurement afterwards. Most wins come from doing less work (fewer queries, allocations, bytes), not cleverer code. Sources: Brendan Gregg (USE method, flame graphs); Gil Tene on coordinated omission; Google SRE book ch. 6 (percentiles, not averages); the benchstat/hyperfine documentation on repeated runs and variance.

Steps

1. Define the number and the target. Metric, percentile, load and environment: "p95 of GET /orders ≤ 200 ms at 50 req/s open-model load", "job wall time", "heap after 10k iterations", "gzipped bundle bytes". Averages hide tails; report p50/p95/p99. Write it down before changing code; the acceptance criterion should be this number. Call search_lore for known hot spots and past decisions.
2. Build a reproducible harness. Use the repo's tooling (autocannon, k6, wrk2, pytest-benchmark, go test -bench, hyperfine, JMH, criterion): fixed inputs, realistic data volume, warm-up, and enough repetitions to see variance (≥ 10 runs or the tool's statistical mode). For latency under load use an open-model / constant-arrival-rate generator (k6 constant-arrival-rate, wrk2 -R): closed-loop tools that wait for each response under-report tail latency (coordinated omission). Record the exact command and machine. Use only tools already installed; never install one (npm/pip installs are hook-blocked; go install, cargo install, uv/pipx and a fetching npx are not blocked but are equally forbidden; node_modules is shared). If a needed tool is missing, say so in the evidence and use what exists.
3. Find the bottleneck resource first. For each resource (CPU, memory, disk, network, connection pools, locks) check utilisation, saturation and errors (USE). A request waiting on a pool or a lock is not a CPU problem.
4. Profile, do not guess. CPU: node --cpu-prof, py-spy record, go tool pprof, perf + flame graph, async-profiler. Memory: heap snapshots / allocation profiles taken at two points to see what grows. Database: query log with timings and per-request query count. Frontend: the bundler's analyzer. Fix the widest frame in the flame graph; if the profile disagrees with your hunch, the profile is right.
5. Fix the top item with the least clever change. Usual order: remove an N+1 (batch or join), add the index the plan wants (the sql-query-performance skill), stop serialising unused data, stream instead of buffering, cache a pure or hot read (the caching-strategy skill), move work off the request path (the background-jobs skill). Algorithmic rewrites and micro-optimisations last.
6. Keep behaviour identical, including under concurrency. Run the full suite. A new cache, batch boundary, memoisation or parallelism is a new code path: test it directly, including concurrent callers (stale reads, lost updates, shared mutable state; the concurrency-and-async skill). Never drop a validation, authorisation check or lock to win time.
7. Measure again under identical conditions and compare distributions, not single runs: same command, data and machine, repeated; use the tool's comparison (benchstat old.txt new.txt, hyperfine with several commands) and report the change with its variance. A difference within noise is not a result. If the target is not met, say so and what the profile shows now.
8. Guard the win with a deterministic check. Prefer assertions that do not flake on shared CI hardware: query count per request, allocation count, bundle-size budget, algorithmic complexity on a fixed input. Wall-clock assertions only with generous margins or on dedicated runners.