analyze

Analyze a coder-eval run

You analyze a coder-eval run and write analysis.md into the target directory. The target path is $ARGUMENTS; when it is empty, resolve the run yourself by following ${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md — discover the run root rather than assuming one, and say which run you picked and how. If you reach it through a latest symlink, confirm that symlink resolves before reading through it.

Do all the reasoning yourself in this session — no sub-agents. Batch your Read calls in a single turn and write the report inline.

The run directory layout and its scope-marker files are described in ${CLAUDE_PLUGIN_ROOT}/reference/run-layout.md — read it first; step 1 relies on those markers.

Step 1 — Determine scope

Inspect the target path:

• task.json directly inside → task scope (single replicate).
• ??/task.json subdirectories but no variant.json → task scope, aggregated over replicates per task_id.
• Contains variant.json → variant scope.
• Contains run.json → run scope. If experiment.json is also present, it is a multi-variant experiment.

If the path contains none of those markers, say which markers you looked for and stop. Do not guess a scope from directory names.

Step 2 — Read the data

Task scope (single): read task.json.

Task scope (aggregated replicates): read every ??/task.json and merge — per-replicate arrays for final_status, weighted_score, iteration_count, duration_seconds, total_token_usage.total_cost_usd, and the union of success_criteria_results keyed by criterion description. Drive recommendations from the aggregate ("3/5 replicates failed criterion X"), never from a cherry-picked replicate.

Variant / run scope with more than 20 tasks: do not read the full task.json files — their iterations arrays are large and only useful per task. Extract a compact summary per task with jq (or python3 if jq is missing):

{
  task_id, final_status, weighted_score, duration_seconds,
  iteration_count, model_used, max_turns_exhausted,
  total_cost_usd:  .total_token_usage.total_cost_usd,
  total_tokens:    (.total_token_usage.input_tokens + .total_token_usage.output_tokens),
  assistant_turns: .total_assistant_turns,
  max_turns:       .task_config.resolved.run_limits.max_turns,
  criteria_count:  (.success_criteria_results | length),
  all_criteria_perfect:
    (.success_criteria_results | length > 0 and all(.[]; .score == 1.0)),
  failed_criteria: [
    .success_criteria_results[]
    | select(.score < .pass_threshold)
    | {criterion_type, description, score,
       error_excerpt: ((.error // .details // "")[0:200])}
  ]
}