eval-harness

Eval Harness

Eval-Driven Development treats evals as the unit tests of AI-assisted work: define expected behavior before implementation, run evals continuously during development, track regressions with every change, and use pass@k for reliability rather than a single pass/fail.

Eval types

Capability evals — test whether the agent can do something new:

[CAPABILITY EVAL: feature-name]
Task: description of what the agent should accomplish
Success Criteria:
  - [ ] Criterion 1
  - [ ] Criterion 2
Expected Output: description of the expected result

Regression evals — confirm a change didn't break existing behavior:

[REGRESSION EVAL: feature-name]
Baseline: SHA or checkpoint name
Tests:
  - existing-test-1: PASS/FAIL
  - existing-test-2: PASS/FAIL
Result: X/Y passed (previously Y/Y)

Grader types

1. Code grader — deterministic checks: grep -q "expected pattern" file && echo PASS; go test ./... -run TestAuth; go build ./.... Prefer this whenever the success criterion is mechanically checkable.
2. Model grader (LLM-as-judge) — for open-ended output: give the model a rubric ("does it solve the stated problem? is it well-structured? are edge cases handled?"), have it emit a 1-5 score with reasoning. Use only where a code grader genuinely can't express the criterion.
3. Human grader — flag for manual review when risk or ambiguity is too high to automate:

[HUMAN REVIEW REQUIRED]
Change: what changed
Reason: why human review is needed
Risk Level: LOW/MEDIUM/HIGH

Order of preference: code grader > model grader > human grader. Deterministic beats probabilistic; probabilistic beats un-reviewed.

Metrics

• pass@k — "at least one success in k attempts." pass@1 is first-attempt reliability; pass@3 is success within 3 tries. Typical target: pass@3 > 90%.
• pass^k — "all k trials succeed," a higher bar for reliability. Use for critical paths where a single silent failure is unacceptable (auth, payments, data migrations): pass^3 = 100% means three consecutive clean runs, not one lucky one.