accelint-eval-architect

Eval Architect

Decides whether and how to add automated evaluation to another skill, recommends the right framework (or honestly recommends none), and scaffolds a maintainable walking-skeleton harness. Defaults to cheap deterministic checks; gates LLM-judge calls behind explicit opt-in.

NEVER Do When Adding Evals

• NEVER reach for an LLM judge before ruling out deterministic verification — judge calls cost money per run, drift across model versions, and introduce false positives. A parser/compiler/schema/test-run check is cheaper, stable, and exact. The decision order in references/framework-matrix.md makes a judge unreachable until determinism is proven insufficient.
• NEVER hand-author golden/expected artifacts — they silently rot when the target's schema changes. (Real bug: ac-to-playwright's PERFECT-AC.plan.json used stale field names and failed a metric for months.) Generate goldens from the live schema/validator at scaffold time.
• NEVER pick thresholds blind — a threshold chosen without a baseline run manufactures a meaningless green checkmark. Scaffold thresholds as record-only, measure a baseline, then set numbers from the observed distribution. See references/calibration.md.
• NEVER scaffold the maximalist metric suite on day one — an unmaintained comprehensive eval is worth less than a maintained walking skeleton. Ship one fixture, one metric, one passing test, one regression test; document the extension path. Exception: detector/review skills ship recall AND false-positive-resistance as an inseparable pair — either alone rewards degenerate behavior (flag everything, or flag nothing).
• NEVER ship a metric without a regression test that proves it can fail — a metric that always passes is decoration. Every metric gets a planted-broken input that drives it below threshold.
• NEVER force a persona×scenario grid onto a single-mode skill — when there is one user and one mode, flat fixtures across an input-quality gradient are clearer. Derive the taxonomy from the skill; don't impose it. See references/test-design.md.
• NEVER leave eval source untracked — results/, .venv/, __pycache__/ are gitignored, so it is easy to orphan the source files alongside them (this happened to the reference impl and required bytecode recovery). Commit eval source before the first run.
• NEVER recommend a Python judge framework for a Node-only skill unless judgment genuinely requires it — house toolchain fit lowers the maintenance barrier. Match the harness to the target's existing package.json/pyproject.toml.
• NEVER evaluate taste-based output with an LLM judge — for visual, creative, or aesthetic output, LLM judges are weaker and less honest than humans. Recommend a structured human-review checklist instead. See references/frameworks/human-review.md.