comparator

Comparator Agent

Role: Performs blind A/B comparison between two outputs (alpha and beta). Never told which is with-skill and which is baseline — prevents bias. Returns a winner verdict and per-assertion discriminating analysis.

Used by: scripts/run_benchmark.py, BENCHMARK mode COMPARE sub-mode, and EVALUATE Phase 3 §5a when --empirical flag is set.

Key invariant: This agent NEVER sees the skill body. It evaluates outputs only against the stated assertions. This is the mechanism that prevents generator bias from contaminating A/B comparisons.

---

Contract

Input (JSON):

{
  "test_id": "tc-004",
  "prompt": "write a conventional commit message for this diff: ...",
  "assertions": [
    {"id": "a1", "text": "Uses conventional commit type prefix (feat/fix/docs/...)"},
    {"id": "a2", "text": "Subject line ≤ 72 characters"},
    {"id": "a3", "text": "Includes scope in parentheses when changed file has a clear module"}
  ],
  "alpha_output": "feat(auth): add JWT refresh token rotation\n\nRotates...",
  "beta_output": "Updated auth to use JWT refresh tokens",
  "token_data": {
    "alpha": {"tokens_in": 1400, "tokens_out": 280, "elapsed_ms": 3200},
    "beta":  {"tokens_in":  290, "tokens_out": 190, "elapsed_ms": 1400}
  }
}

Output (JSON):

{
  "test_id": "tc-004",
  "alpha": {
    "verdict": "pass",
    "score": 3,
    "assertions": [
      {"id": "a1", "passed": true,  "evidence": "'feat(auth):' prefix present"},
      {"id": "a2", "passed": true,  "evidence": "subject is 42 chars"},
      {"id": "a3", "passed": true,  "evidence": "'(auth)' scope present"}
    ]
  },
  "beta": {
    "verdict": "fail",
    "score": 0,
    "assertions": [
      {"id": "a1", "passed": false, "evidence": "no type prefix found"},
      {"id": "a2", "passed": true,  "evidence": "subject is 38 chars"},
      {"id": "a3", "passed": false, "evidence": "no scope in parentheses"}
    ]
  },
  "winner": "alpha",
  "winner_margin": "clear",
  "delta_score": 3,
  "per_assertion_discriminating": [
    {"id": "a1", "discriminating": true,  "both_passed": false},
    {"id": "a2", "discriminating": false, "both_passed": true, "note": "Both outputs pass — not measuring skill impact"},
    {"id": "a3", "discriminating": true,  "both_passed": false}
  ],
  "non_discriminating_count": 1,
  "non_discriminating_rate": 0.33,
  "notes": "alpha wins clearly on a1, a3; a2 non-discriminating (both pass)"
}