discriminating-experiment-design

Note: the crate paths, documents, and measured numbers below are illustrations from the aurora-agent workspace where these methods were developed and tested. The methods themselves apply to any experiment or benchmark design.

Discriminating experiment design

A benchmark on which every serious method gets the same answer has measured the benchmark, not the methods. The workspace hit this on its own flagship comparison and turned the recovery into a repeatable method.

First, notice that your benchmark cannot discriminate

On the shipped reference world, the compiler under test was matched exactly — the identical eleven-fact selection, not merely the same count — by a tuned graph walk, a lexical retriever, and a directed dependency walk (docs/FINDINGS.md). The findings document says what that means, and yours should too: "That is partly a property of the benchmark, not of the methods." The reference world sat at one corner of the structural space: distractors on a hub leaf, no relay chain, tags that name the answer. Any method that exploits any of adjacency, lexical overlap, or dependency structure lands on the same answer there.

Symptoms to check for on your own benchmark: identical selections (not just identical scores), one structural feature that every method can exploit, and a winner that flips when a single world-construction choice changes.

Construct a family that could discriminate

Turn the properties your benchmark accidentally fixed into explicit parameters, holding the decisive content and the judging oracle constant. The workspace's generator (crates/worldgen) made three properties into knobs:

• Attachment — are distractors attached at a hub or near the target?
• Relay depth — is decisive evidence behind a chain of intermediate steps?
• Tag camouflage — do distractor labels tokenize into the protected vocabulary, or are they lexically distinct from it?
• plus distractor count as a scale axis.

Each axis is aimed at one method family's known crutch: attachment and relay depth break undirected adjacency; camouflage breaks character-level similarity (camouflaged tags share most of their trigrams with the query — exactly the similarity a hashed-trigram basis rewards, which is why the embedding retriever's closure fell to 36% where BM25 held 91%). Design each knob to break a named assumption, and predict in writing which family it should break before running.