compare-runs

Compare Runs

Overview

Retort is a Design of Experiments engine — every run sits at a point in the factor space, and replicates exist to measure noise. Unlike pourpoise's compare-attempts (which maintains a ranked leaderboard of ad-hoc submissions), comparing retort runs is about factor effects and qualitative divergence, not "who won".

This skill is the qualitative counterpart to retort analyze / retort report effects. The ANOVA tells you whether a factor is significant. This skill tells you what changed in the generated code as you varied it.

Parameters

• experiment_dir (required): e.g. experiment-1/. Must contain retort.db and a runs/ directory of per-run archives, each already evaluated by evaluate-run (i.e. each rep<N>/ has an evaluation.md and findings.jsonl).
• output_file (optional, default: {experiment_dir}/reports/comparison.md): Where to write the comparison report.
• group_by (optional, default: all factors): Comma-separated factors whose effect you want to isolate. The other factors are aggregated over.
• include_failed (optional, default: true): Whether to include failed runs in the comparison (they're still informative — failure itself is a finding).

Steps

1. Discover evaluated runs

find {experiment_dir}/runs -name evaluation.md -not -path "*/salvaged-*/*" | sort

Parse each path to extract the cell (from the parent directory name, e.g. language=rust_model=opus_tooling=beads) and replicate (from the rep<N> or rep<N>-failed segment).

Constraints:

• You MUST skip runs under salvaged-*/ — those are manually-archived artifacts from before auto-archival existed, and their layout is not regular.
• You MUST handle both rep<N>/ (success) and rep<N>-failed/ (failure) directory names.
• You SHOULD warn but not fail if some cells have no evaluation (evaluator hadn't run yet).

2. Load findings and metrics

For each evaluated run, load:

• Factors (from the cell directory name or the run's stack.json)
• Summary metrics (from evaluation.md header: requirements pass count, test counts, build status)
• Findings (from findings.jsonl) — categorize by kind and severity
• Architecture signal (from summary/index.md: the "Shape" one-liner)