agent-wiki-compare-outcomes

Agent Wiki — Compare Outcomes

Overview

Use this pass after summarize/extract/synthesize when there are multiple trajectories that can be judged as successful or failed. It can judge outcomes with an LLM from the normalized transcript, so it does not need to depend on benchmark-specific success/failure labels. It derives contrastive guidelines: rules that are supported by a failed path, a successful path, and concrete evidence from task wording, tool/API documentation, tool/API calls, transcript evidence, optional failure snippets, and optionally an LLM success/failure judgment.

This pass exists to avoid hand-authored domain knowledge. Do not write a rule just because you know the benchmark or application. Write a rule only when the input trajectories contain the evidence.

Workflow

Step 1: Build an Evidence Pack

Run the bundled script over normalized trajectory JSON files:

uv run python explorations/agent-wiki/skills/agent-wiki-compare-outcomes/scripts/compare_outcomes.py \
  --input <normalized-dir-or-json> \
  --out-json <analysis.json> \
  --out-md <analysis.md> \
  --judge-outcomes always

Pass --input multiple times to compare several experiment arms.

The script groups traces by metadata.task_id when present; otherwise it uses a normalized task request. For each group it compares successful and failed runs, then extracts:

• task request text;
• stored outcome and failure snippets when present;
• LLM-judged outcome when --judge-outcomes missing or --judge-outcomes always is set;
• observed tool/API calls from stats.top_tools, code snippets, and source api_calls.jsonl when available;
• tool/API descriptions shown in the trajectory transcript.

Judging modes:

• --judge-outcomes never: use only stored outcome.success.
• --judge-outcomes missing: judge only traces without stored outcomes.
• --judge-outcomes always: ignore stored success labels and use the LLM judgment for all traces.

Prefer --judge-outcomes always when the available stored labels come from a benchmark evaluator or another dataset-specific schema. Use stored outcomes only when they are trusted, dataset-neutral annotations you are comfortable using as ground truth.