analyzer

Analyzer Agent

Role: Synthesizes results across multiple test cases from a benchmark run. Identifies patterns in failures, detects non-discriminating assertions, flags high-variance cases, and generates prioritized improvement recommendations.

Used by: scripts/run_benchmark.py (post-aggregation analysis step), BENCHMARK mode step 6 (refs/modes/benchmark.md), and /opt --from-benchmark.

Input: the benchmark.json produced by aggregate_benchmark.py. Output: structured analysis JSON + human-readable recommendation list.

Key invariant: The Analyzer NEVER re-grades individual test cases — it only synthesizes already-graded Comparator outputs. This preserves the separation between grading and analysis.

---

Contract

Input (full benchmark.json object — see scripts/aggregate_benchmark.py):

{
  "skill": "git-diff-summarizer",
  "skill_version": "1.2.0",
  "timestamp": "2026-04-19T10:00:00Z",
  "mode": "real | simulated",
  "summary": {
    "total_cases": 10,
    "pass_rate": 0.70,
    "baseline_pass_rate": 0.40,
    "delta_pass_rate": 0.30,
    "avg_tokens_with_skill": 1840,
    "avg_tokens_baseline": 920,
    "token_overhead": 920,
    "token_overhead_pct": 100.0,
    "avg_latency_with_ms": 3200,
    "avg_latency_base_ms": 1600,
    "non_discriminating_rate": 0.22,
    "variance": 0.18
  },
  "per_case": [ ... ]
}

Output (JSON):

{
  "skill": "git-diff-summarizer",
  "verdict": "BENCHMARK_PASS | BENCHMARK_MARGINAL | BENCHMARK_FAIL",
  "verdict_reason": "delta_pass_rate=0.30 exceeds 0.15 threshold; token overhead 100% is HIGH",

  "patterns": {
    "top_failure_modes": [
      {
        "description": "ZH inputs not triggering skill-specific formatting",
        "affected_cases": ["tc-003", "tc-007"],
        "frequency": 0.20,
        "dimension": "D7 Metadata / triggers",
        "severity": "HIGH"
      },
      {
        "description": "Error handling section not followed on empty input",
        "affected_cases": ["tc-005"],
        "frequency": 0.10,
        "dimension": "D4 Error Handling",
        "severity": "MEDIUM"
      }
    ],
    "non_discriminating_assertions": [
      {
        "assertion_text": "The response is not empty",
        "appears_in_cases": ["tc-001", "tc-002", "tc-004"],
        "rate": 0.30,
        "recommendation": "Replace with: 'Response uses the §3 Workflow phase headers defined in the skill'"
      }
    ],
    "high_variance_cases": [
      {
        "test_id": "tc-008",
        "note": "alpha won by 1 assertion; marginal win — result may flip on re-run",
        "recommendation": "Make assertion a3 more specific to reduce ambiguity"
      }
    ],
    "token_assessment": {
      "overhead_pct": 100.0,
      "verdict": "HIGH — skill doubles token cost",
      "recommendation": "Apply S15 Skill Body Slimming. Target: trim workflow sections to table format (-150 tokens). Current est_tokens_p50 should be declared in production: YAML block."
    }
  },

  "recommendations": [
    {
      "priority": 1,
      "type": "fix_failure",
      "action": "Add ZH trigger phrases for the skill's primary output mode",
      "strategy": "S9 (Metadata / trigger coverage)",
      "expected_delta": "+0.10 pass_rate",
      "effort": "LOW"
    },
    {
      "priority": 2,
      "type": "fix_failure",
      "action": "Add empty-input guard at top of §4 Workflow",
      "strategy": "S5 (Error Handling expansion)",
      "expected_delta": "+0.05 pass_rate",
      "effort": "LOW"
    },
    {
      "priority": 3,
      "type": "reduce_cost",
      "action": "Apply S15: compress §4–§6 workflow tables, trim examples to 2",
      "strategy": "S15 (Skill Body Slimming)",
      "expected_delta": "-40% token_overhead",
      "effort": "MEDIUM"
    },
    {
      "priority": 4,
      "type": "improve_evals",
      "action": "Replace 3 non-discriminating assertions with skill-specific checks",
      "strategy": "eval improvement (see non_discriminating_assertions above)",
      "expected_delta": "more reliable benchmark scores",
      "effort": "LOW"
    }
  ],

  "optimize_command": "/opt --from-benchmark benchmarks/2026-04-19T10:00:00Z/benchmark.json",
  "retest_command": "python3 scripts/run_benchmark.py --skill my-skill.md --cases test-cases.json --compare-version v1.2.0"
}