bench-check

/bench-check — Performance Regression Check

Run the project's benchmark suite, compare results against a saved baseline, flag regressions beyond a threshold, and optionally update the baseline. Prevents silent performance degradation between releases.

Arguments

• $ARGUMENTS may contain:
  • --save-baseline — run benchmarks and save as the new baseline (no comparison)
  • --compare-only — compare against baseline without updating it
  • --threshold N — regression threshold percentage (default: 15%)
  • No arguments — compare against baseline, then update it if no regressions

Phase 0 — Pre-flight

1. Confirm we're in the codegraph repo root
2. Check that benchmark scripts exist:
   • scripts/benchmark.js (build speed, query latency)
   • scripts/incremental-benchmark.js (incremental build tiers)
   • scripts/query-benchmark.js (query depth scaling)
   • scripts/embedding-benchmark.js (search recall) — optional, skip if embedding deps missing
3. Parse $ARGUMENTS:
   • SAVE_ONLY=true if --save-baseline
   • COMPARE_ONLY=true if --compare-only
   • THRESHOLD=N from --threshold N (default: 15)
4. Check for existing baseline at generated/bench-check/baseline.json
   • If missing and not --save-baseline: warn that this will be an initial baseline run

Phase 1 — Run Benchmarks

Run each benchmark script and collect results. Each script outputs JSON to stdout.

1a. Build & Query Benchmark

output=$(timeout 300 node scripts/benchmark.js 2>&1)
exit_code=$?

If exit_code is 124: record "timeout" for this suite and skip to the next suite. Else if exit_code is non-zero: record "error: $output" for this suite and skip to the next suite.

Extract:

• buildTime (ms) — per engine (native, WASM)
• queryTime (ms) — per query type
• nodeCount, edgeCount — graph size

1b. Incremental Benchmark

output=$(timeout 300 node scripts/incremental-benchmark.js 2>&1)
exit_code=$?

If exit_code is 124: record "timeout" for this suite and skip to the next suite. Else if exit_code is non-zero: record "error: $output" for this suite and skip to the next suite.