evals-router

Evals Router

Route LLM/RAG eval work through the smallest dependency-ordered proof plan. Start with the proof closest to the changed claim and keep package, scenario, scorer, runtime, observability, hosted, and external authorities separate.

When To Use

Use for eval coverage, failed traces, judge prompts, labels, RAG evidence, synthetic cases, evaluator profiles, scenario contracts, or review tooling. Do not use for ordinary software test implementation.

Inputs

Need the user goal, claim being evaluated, candidate identity, and available traces, labels, scorecard, prompt, retrieval evidence, scenarios, receipts, or target artifacts.

Discover repository-owned evidence before asking: inspect current source, schemas, reports, receipts, runtime state, and user-named artifacts. Ask one question only when this inspection cannot resolve a choice that would materially change the route.

For release-impacting or multi-lane work, also inventory the candidate revision or digest, scenario set, rubric version, runner, execution profile, provider/model or deterministic identity, generated state, and authority of each receipt.

Outputs

Primary artifacts include a patched eval, judge prompt, trace analysis, synthetic case, RAG check, review-interface spec, eval-contract repair, or blocked report. Add only the receipts needed to support the user's claim.

Artifact shapes:

• eval-audit: claim_id -> case_id|gap_id
• error-analysis: trace_id | failure_mode | owner | rerun_command
• evaluate-rag: sentence_id -> supported chunk_ref | unsupported
• multi-lane evidence: lane | claim | evidence_ref | identity | command | status | proves | does_not_prove | owner | next_check

Use pass, fail, blocked, or not_run per lane. Do not collapse mixed lane results into one aggregate verdict.

Workflow

1. Build the smallest dependency-ordered route plan that can answer the user's claim. Name each lane, evidence source, owner, proving command, and authority. Execute one lane at a time without allowing one lane to prove another.
2. Inventory evidence as present, stale, malformed, inaccessible, generated_but_untrusted, or absent. Resolve repository-owned discovery paths before asking for missing inputs.
3. Run the deterministic proof closest to the changed claim. Produce the smallest checkable artifact: claim-to-case map, failure table, binary judge, calibration bundle, synthetic case, sentence-support map, or review schema.
4. If the focused check fails, classify ownership, patch only the responsible prompt, case, judge, retrieval evidence, report, validator, or pipeline contract, and rerun the same check before widening.
5. When a broad check fails after focused proof passes, compare the same command and assumptions against a clean or known-good baseline when safe and practical. Classify the result as introduced, pre_existing, environment, permission, toolchain_runtime, or unresolved.
6. Widen only to dependent package, scenario, scorer, runtime, observability, hosted, or external lanes. Preserve independently proved upstream results when another lane blocks.
7. Before staging or running Tessl, classify the scenario as package_scored_fixture or response_producing_scenario. Align task text, criteria inputs, runner capability, staged paths, and expected artifacts. Assign failures to task, criteria, runner, staging, or pipeline_guardrail before patching.
8. When preview/apply or dry-run/write modes exist, require a non-blocking preview, review the proposed paths, apply only with mutation authority, and rerun consumer validation against the applied artifact.
9. Snapshot relevant generated and untracked state before eval tooling. After the run, classify new artifacts as required evidence, expected cache, temporary staging, or unintended residue; remove only residue created by the current run.
10. Reconcile lane statuses, exact command evidence, blockers, and the smallest next check. Never invent a score or promote fallback evidence into proof of a blocked lane.