running-benchmarks

Running Benchmarks

Overview

You are the machine the producer ran so a perf regression wouldn't reach a human — or production — first. Benchmarks are backpressure: they say "no" to a slow patch at the boundary where it was introduced, not three releases later when someone files a latency bug.

Core principle: a benchmark number is meaningless on its own. A single "after" run proves nothing. A verdict requires a baseline to compare against and a way to separate a real change from run-to-run noise. Everything below exists to enforce those two things, because the natural temptation is to run once and eyeball it — and that is worthless.

When to Use

• A backpressured loop is on a performance-sensitive project and needs to confirm a patch didn't regress perf — a quick targeted check each iteration, and the full suite before leaving the loop.

Not for: projects with no benchmark suite or no perf sensitivity — skip benchmarking entirely rather than inventing it. This is not a correctness check; tests cover that.

Step 1 — Discover the suites and their time budgets

If the project has a BACKPRESSURE.md, check it first — it may name the suite(s) and exact commands to use. Otherwise find the commands yourself — package.json scripts, Makefile, a benchmarks//bench/ dir, CI config, README. Identify which suites exist and roughly how long each takes. Expect a spread: fast micro/targeted suites (seconds) and full end-to-end suites (minutes). Note what each one actually measures so you can map a suite to the code path you touched.

Step 2 — Pick the suite for the moment

• Each iteration: run the fast, targeted suite that covers the path you changed. It must be cheap enough to run after every patch without rabbit-holing.
• Before done: run the full suite once. Targeted-green is necessary, not sufficient — a change can help one path and regress another.

Don't run an 8-minute end-to-end suite to check a change that only touches one micro path; don't declare done off a micro suite alone.