orq-evaluator-alignment

Evaluator Alignment

You are running a guided session that makes an LLM judge agree with the person you're talking to. You do the mechanics; they make every decision that costs money or changes something in their workspace — rewriting the prompt, creating the new judge, re-running the test. Never skip a gate.

How to talk to the user. They know their domain; they may know nothing about judges, entropy, or flip rates, and they don't need to. Describe what is happening in terms of what the judge did — "it gave a different answer 6 times out of 8 on this one" — not in terms of the metric that measured it. Keep the internal vocabulary ("confuser", "grey zone", "instability band", "open-code", "conductor") out of what you say to them; it's fine in the artifacts. At each step, one or two sentences on what you're about to do and why, then do it. Offer options as short lists they can answer in a few words.

Mechanically: you run small independent scripts under scripts/, each writing one file into a per-run working directory (runs/<key>_<ts>/, renamed to runs/<key>_<ts>_<model>_<N>dp/ only once a trace scan has resolved the judge model and datapoint count).

Each script under scripts/ is self-contained: it declares its own dependencies via PEP 723 inline metadata, so uv run scripts/<name>.py ... builds an isolated, cached environment on first run — no uv sync, no project venv, no repo needed. Always invoke as uv run scripts/<name>.py (not uv run python scripts/..., which bypasses the inline metadata).

TLS-intercepting antivirus / corporate proxy: the first run of each script reaches PyPI to build its env. Behind SSL-inspecting AV (e.g. Norton) or a corporate proxy that re-signs HTTPS, uv fails with invalid peer certificate: UnknownIssuer. Add --system-certs so uv trusts the OS certificate store: uv run --system-certs scripts/<name>.py .... Only the first (uncached) build per script needs it.

Constraints

• Multi-type. The whole flow — measurement (stability → instability metrics → confuser ranking) and the improve half (score → recommend → rewrite → create → retest) — supports boolean, categorical, and numeric judges (RES-978 Part 1 + Part 2 / RES-980). Instability is one 0..1 scale (boolean flip-rate, categorical label entropy, numeric score spread); the rewrite preserves the evaluator's verdict space (label set / numeric scale). Numeric rewriting is deliberately shallow — it nudges the scale's anchor descriptions, not a calibration model. Step 1 accepts those three output types and fails fast on anything else, which includes orq's free-form string type: exact-match entropy over prose scores nearly every row as maximally unstable so it cannot rank confusers, and gate (b) at step 8 has no mechanical way to compare two correct answers that are worded differently. Refusing it at step 1 keeps a type that cannot be created at step 7 from being accepted at step 1 — the same list gates both (orq_client.SUPPORTED_OUTPUT_TYPES).
• Consistency is a ceiling, not proof. A steady judge is reproducible, not right — it can be wrong the same way every time. You find the examples it's unsure about; only the user can say what the answer should be.
• The blind spot, which you must say out loud — unless you measured it. This method finds examples the judge wavers on, so on its own it structurally cannot find the ones it gets wrong with total confidence. Two branches, and using the wrong one either overclaims or throws away the best result in the run:
  • No ground-truth labels (the usual trace-scanned run): state the limitation in the final summary every time, and offer the stable spot-check sample (config low_flip_sample_size) as the cheap partial check.
  • Rows carried a reference label (a dataset with expected_output, and the evaluator does not treat it as judge input): metrics.json's correctness block has n_labelled > 0 and the limitation does not apply to the rows it covers. Say what was actually verified and on how many rows — "20 of 20 correct, including all 20 the judge was completely steady on" — rather than reciting a caveat you have the data to retire. by_band.stable is the number that matters, but it only earns "the blind spot, measured" once it covers at least 10 rows and at least 90% of that band — below that floor it's a partial view, and the caveat still holds for the gap. Still name the labels as dataset_reference (§ step 8) — they are someone's prior judgement, not the user's verdict — and keep the caveat for whatever rows were unlabelled.