grader

Grader Agent

Role: Independent grader for skill-writer evaluations. Runs as a subagent in a fresh context so it cannot see the generator's reasoning — this is the only mechanism in skill-writer that actually breaks generator bias.

Invoked by: scripts/run_trigger_eval.py, scripts/optimize_description.py, the /eval mode when it spawns parallel subagents, and the /benchmark mode for A/B comparative grading.

v3.5.0: Extended with grading_mode field to support three grading modes: single (original), comparative (A/B delta), and discriminating_check.

---

Contract

Mode 1 — Single Output Grading (original)

Input:

{
  "grading_mode": "single",
  "test_id": "tc-003",
  "prompt": "summarize this git diff for a PR description",
  "expectation_type": "should-trigger | should-not-trigger | behavioral",
  "skill_name": "git-diff-summarizer",
  "skill_description": "...",
  "assertions": [
    {"id": "a1", "text": "The response mentions files changed"},
    {"id": "a2", "text": "The response groups changes by feat/fix/refactor"}
  ],
  "with_skill_output": "...",
  "baseline_output": "..."
}

Output:

{
  "grading_mode": "single",
  "test_id": "tc-003",
  "triggered": true,
  "expectations": [
    {"id": "a1", "text": "...", "passed": true,  "evidence": "response says 'Files changed: 3'"},
    {"id": "a2", "text": "...", "passed": false, "evidence": "no grouping present"}
  ],
  "verdict": "partial",
  "notes": "with-skill output meets 1/2 assertions; baseline meets 0/2"
}

Mode 2 — Comparative A/B Grading (v3.5.0 — BENCHMARK mode)

Grades both outputs independently then computes delta. Never told which is "with-skill".

Input:

{
  "grading_mode": "comparative",
  "test_id": "tc-003",
  "prompt": "summarize this git diff for a PR description",
  "assertions": [
    {"id": "a1", "text": "The response mentions files changed"},
    {"id": "a2", "text": "The response groups changes by feat/fix/refactor"}
  ],
  "output_alpha": "...",
  "output_beta": "...",
  "token_data": {
    "alpha": {"tokens_in": 1200, "tokens_out": 340, "elapsed_ms": 3100},
    "beta":  {"tokens_in":  280, "tokens_out": 410, "elapsed_ms": 1700}
  }
}