geniro-claude-harness: Skill for Claude Code

eval-loop — drive the module improvement cycle

Contents

• Iron rules (executor pick · money asks first · holdout dark · one change per experiment · rubric version bump)
• Intake
• The cycle — 1. Error analysis → EXP file · 2. Screen · 3. Judge (free path) · 4. Verdict · 5. Confirm · 6. Promote
• Side jobs — calibrate · add-task · new module · A-vs-A
• Reporting

Working dir for everything here: evals/loop/. Read its DESIGN.md on first use in a session. Mechanics live in the scripts; this skill supplies the judgment the scripts deliberately do not encode.

Iron rules (hold at every step)

• The executor is the user's pick, every time it is a pick. Two adapters can serve a run — adapters/claude-subagent.md (in-session subagents, free) and adapters/cursor-cli.sh (cursor-agent, paid). Whenever a step could run on either, ask which with AskUserQuestion before launching, naming what each costs and listing the free one first. Ask per step, not per session: a pick made for an earlier step is not consent for the next one, and a run that does not need the paid executor should never take it by default. Only a step one adapter cannot serve skips the question — say which adapter and why.
• Money asks first. Before ANY paid sweep: run run.sh --probe, put the extrapolated cost in a chat message, and get an AskUserQuestion approval naming the dollar figure, the executor adapter, AND the model. The per-sweep hard ceiling is run.sh --max-usd (default $50) — raise it only with the user's number. Never launch on a stale rate — after a change to task shape, workspace size, or model, the probe is mandatory (adapters/cursor-prices.json §rule). Judging via Claude subagents is free; the --phase judge CLI fallback is paid and needs the same approval.
• Holdout stays dark. Never open holdout task/rubric content while a variant is being tuned. loop.sh confirm runs it; you read only its scores.
• One change per experiment, named in an EXP file BEFORE the screen run, with the prediction written down. A tie is a result — record it.
• Rubric edits bump version (integer, any edit) — and invalidate every standing baseline run for that task; re-sweep the champion before comparing. Carve-out: acceptance_evidence is non-scoring provenance — an edit touching only it does not bump.
• Read transcripts before trusting numbers. No verdict is reported to the user until you have opened at least the failing trials' findings and one judge verdict and confirmed the failures look fair.
• Every choice goes through AskUserQuestion. The spend approval above, the §1 run/edit/drop call, the §4 confirm/iterate/stop call, the §6 promote call, and the calibrate / add-task row walks are this cycle's gates, not the complete set — a pick that arises anywhere else still routes through the tool (skills/_shared/gate-rendering.md §Lean-question conventions owns the rule).
• Committed benchmark content is anonymized. A task mined from a private repository never carries, in any committed file: the repo/company/product name, tracker ticket IDs, PR/issue numbers (including in task ids — real-N, never pr-1234), usernames or people's names, email domains, review-comment IDs, or machine-local paths. Its repo location goes through repo_alias + the gitignored repos.local.json instead. A task staged directly against a known-public repository (e.g. the spec-check module's OSS fixtures) commits repo_url — the URL discloses nothing a clone can't already see. Commit SHAs and in-repo file paths are fine either way (needed for staging and matching). Before committing a private-repository task, grep it for the source repo's name, the ticket prefix, the author handles, PR #, and long digit runs (comment IDs) — this repo is public, the private benchmark sources are not.