nexus-eval-harness

Nexus Eval Harness

Context: the harness under tests/eval/ drives the real production path — StreamingOrchestrator plus tool continuation — with a mock or live tool executor swapped in, and grades the tool calls a model emits. It is a fixture system, and almost every surprising result is the fixture talking, not the model. This skill owns the fixtures, the configs, the executors and the reports.

Workflow

1. Pick the job and open its protocol. Work from the protocol; this router names procedures, it does not contain them.

   Job	Protocol
   Add a scenario, or fix one that grades wrongly	protocols/add-a-scenario.md
   Write or change a config, choose targets, mode, retries	protocols/configure-a-run.md
   A run produced nothing, all-fails, a hang, or odd numbers	protocols/debug-a-run.md
   Change the executors, assertions, loader or reports	protocols/extend-the-harness.md

2. Derive every list from the tree, never from this skill. It names no scenarios, no configs, no models and no env-var table on purpose, and you MUST NOT add one — the harness gains knobs faster than a document survives.

   ls tests/eval/scenarios/ tests/eval/configs/
   grep -rhoE "get(Number|List)?Env\('[A-Z_]+'\)|process\.env\.[A-Z_]+" tests/eval/ \
     | grep -oE "[A-Z][A-Z_]{3,}" | sort -u    # every knob, including the
                                               # ones ConfigLoader mediates
   

3. Before calling any scenario change done, run the checker from the repo root and fix everything it prints:

   python3 .claude/skills/nexus-eval-harness/scripts/check_scenarios.py
   

4. NEVER trust jest's exit code as the verdict on a run, and never report a pass rate you read from stdout. The saved reports under the configured artifacts dir are the only source of truth, and they are written even when the run times out — see references/run-behavior.md.

5. At the end of a session that used this skill, run protocols/self-refine.md.