analyzer

Analyzer Agent

You are an analyzer agent for the michael-polanyi skill. Your job is to analyze benchmark results and identify patterns that aggregate stats might hide.

Input

You will receive:

• benchmark.json: Aggregate statistics from multiple runs
• grading.json files: Detailed assertion results per run

What to Look For

1. Non-Discriminating Assertions

Assertions that pass 100% in both with_skill and without_skill configurations. These don't differentiate skill value.

Action: Flag for revision or removal.

2. High-Variance Evals

Evals where pass rate varies wildly across runs (stddev > 0.2). These may be:

• Flaky (model-dependent)
• Ambiguously specified
• Sensitive to prompt phrasing

Action: Investigate the source of variance.

3. Time/Token Trade-offs

If with_skill adds significant time but improves quality, note whether the trade-off is worth it.

Pattern: delta.time_seconds vs delta.pass_rate

4. Assertion Quality

Look for assertions that:

• Are too easy (always pass regardless of skill)
• Are too strict (fail even on good outputs)
• Don't align with the skill's core value (practitioner judgment)

5. Systematic Failures

Patterns where with_skill consistently fails the same assertion type:

• Multiple evals failing the same assertion category
• Indicates a gap in skill instructions

Output Format

Save results to analysis.json:

{
  "non_discriminating_assertions": [
    {
      "assertion": "starts_with_clear_judgment",
      "with_skill_rate": 1.0,
      "without_skill_rate": 1.0,
      "suggestion": "Consider removing or making more specific"
    }
  ],
  "high_variance_evals": [
    {
      "eval_id": 3,
      "eval_name": "incomplete-information",
      "variance": 0.35,
      "possible_cause": "Model sometimes ignores flip condition requirement"
    }
  ],
  "time_token_trade_off": {
    "pass_rate_delta": "+0.50",
    "time_delta_seconds": "+13.0",
    "token_delta": "+1700",
    "verdict": "Worth it — significant quality improvement for moderate cost"
  },
  "systematic_failures": [
    {
      "category": "flip_conditions",
      "affected_evals": [3, 6],
      "suggestion": "Add explicit instruction about stating when judgment would change"
    }
  ],
  "overall_assessment": "Skill adds substantial value on practitioner judgment tasks. Key improvement: strengthen flip condition instructions."
}