equal-engineering-baselines

Note: the crate paths, documents, and measured numbers below are illustrations from the aurora-agent workspace where these methods were developed and tested. The methods themselves apply to any comparison or benchmarking effort.

Equal-engineering baselines

A comparison is only evidence if the baselines were engineered as seriously as the system being sold. The workspace learned this from its own upstream distribution, and encoded the lesson in a harness (crates/baseline) whose design decisions are worth copying anywhere.

Tune the baseline where it wins, not where it loses

The distribution's own comparison script measured its graph baseline at depth 7 and unbounded only — the two settings where the walk returns the entire 761-fact world. It never measured depths 5 or 6, where the identical code returns 11 facts, matching the compiler exactly. The published comparison therefore showed a 69x advantage that disappears entirely under equal tuning (docs/FINDINGS.md, "The distribution's own baseline script is a strawman").

The method:

1. Sweep every tuning knob the baseline has — depth for walks, k for retrievers — and report the baseline at its best setting, not at the setting that flatters your system.
2. Give each family its strongest member. The workspace's panel runs graph walks at four depths, a connected-component upper bound, two lexical budgets, two embedding budgets, and an unbounded directed walk (crates/baseline, default_panel()), alongside full-context as the recall ceiling.
3. Label proxies as proxies. The panel's "embedding" retriever is a fixed-basis hashed-trigram model and every report line says so: "a lexical proxy for embedding retrieval, not a neural model." A baseline that impersonates a stronger method inflates your win over the real thing.
4. If a baseline stays competitive under equal optimization, report that result. On the reference world, three differently-engineered baselines select the identical eleven facts the compiler selects — not the same count, the identical set — and the comparison document says the cheapest admissible strategy is the graph walk, not the compiler.