performance:benchmark

Benchmark

Goal: a comparison the user can trust. Every arm runs the same scenario from the same starting state, the arms are interleaved so machine drift hits them equally, the noise floor is measured before any difference is read, and a difference counts only when it clears that floor.

Tools

• hyperfine 2.0 or later: !hyperfine --version 2>/dev/null || echo "not found. Required unless the project's own tooling covers its job."
• bun (runs compare.ts): !bun --version 2>/dev/null || echo "not found. Required for compare.ts."

Existing Tooling

Look for the project's own benchmarks before building one: package scripts, a bench/ directory, benchmark tests, CI jobs that time something. Check each against the requirements the setup below is built to meet:

• Scenario. It runs the scenario the user cares about, from the same starting state every run.
• Isolation. Network, remote state, and caches are stubbed, held fixed, or reset per run.
• Interleaving. Arms alternate within one run or one process, so drift hits every arm.
• Samples. It keeps every run's measurement, enough of them to show an A/A noise floor, not only a mean.
• Significance. It reports a significance test, or its samples can feed compare.ts report.

Use the existing tooling when it meets every requirement. When it misses some, compose around it and keep its scenario: run its command as a compare.ts arm for interleaving and significance, or add a --prepare reset. Tell the user which requirement it missed and what you added.

Scenario

Pin the scenario before timing it. A run that changes its own input measures a different scenario each time.

• Reset state per run with --prepare (restore a fixture, clear or warm a cache), and one-time setup with --setup. Match the cache state to the scenario the user cares about. A cold-start metric needs a cold cache on every run.
• Stub or hold fixed everything outside the program: network, remote state, other processes' files. When the program syncs with a remote, point it at a local copy that stays unchanged.
• Pass -N when the command is a single executable with arguments, which removes the shell's startup from every sample.
• Before a baseline, put the machine on AC power with power saving off, and quiet it as far as it goes. When it cannot go idle, interleaving keeps the comparison fair. compare.ts records the load average around each round and flags rounds above twice the core count.
• When the program is a test suite, raise the runner's timeout in every arm's command (bun test --timeout, pytest --timeout), so a load spike slows a run instead of failing it. Read the first timeout before raising it, since it can be a real cost worth a candidate.
• For a shell script or shell startup (.zshrc, .bashrc), read references/shell.md.