compare-tasks

Compare two harbor runs on one task

Use when given two harbor run names and a task name, and the goal is to understand why the two runs differ on that task — not just that they differ.

Inputs

• RUN_A: harbor run name (e.g. sonnet46-full)
• RUN_B: harbor run name (e.g. pi-sonnet46-full)
• TASK: bare task name (e.g. extract-elf, not terminal-bench/extract-elf)
• RUNS_DIR: defaults to evals/harbor/runs/ relative to the repo root

Procedure

1. Find each run's trial directory for the task

Harbor 0.8 names trial dirs <task>__<random-suffix> (e.g. extract-elf__bU3GHs4), not <task>.1. The suffix is unique per trial, so don't guess it — discover it from disk:

TRIAL_A_DIR=$(ls -d "$RUNS_DIR/$RUN_A/${TASK}__"*/ 2>/dev/null | head -1)
TRIAL_B_DIR=$(ls -d "$RUNS_DIR/$RUN_B/${TASK}__"*/ 2>/dev/null | head -1)

If either is empty, that run didn't include this task — stop and say so. (ls "$RUNS_DIR/$RUN_A/" shows what's there.)

If you want to confirm the match, every result.json carries task_name and trial_name:

jq '{task_name, trial_name}' "$TRIAL_A_DIR/result.json"

2. Headline facts

The fastest path is to let cmd.py task do it for you — it already prints status, reward, duration, tokens, turns, cost, error class, and the tail of the verifier stdout:

./evals/harbor/cmd.py task "$RUN_A" "$TASK"
./evals/harbor/cmd.py task "$RUN_B" "$TASK"

Only drop to raw jq against result.json if you need a field cmd.py task doesn't print. The actual shape (harbor 0.8 TrialResult):

jq '{
  reward: (.verifier_result.rewards.reward
           // (.verifier_result.rewards | to_entries | .[0].value)
           // null),
  rewards_all: .verifier_result.rewards,
  duration_seconds: ((.finished_at | fromdateiso8601) - (.started_at | fromdateiso8601)),
  input_tokens: .agent_result.n_input_tokens,
  cache_tokens: .agent_result.n_cache_tokens,
  output_tokens: .agent_result.n_output_tokens,
  cost_usd: .agent_result.cost_usd,
  error_type: .exception_info.exception_type,
  error_message: (.exception_info.exception_message // "" | split("\n")[0])
}' "$TRIAL_A_DIR/result.json"