analyze-results

Analyze results & make figures

Needs imported results under .research/results/ in a clone.

1. Produce the metrics mechanically with airas-eval. Metrics are never computed by hand: the template ships a Makefile wired to airas-eval, a trusted, versioned scoring layer that computes a fixed metric set per task type from the raw predictions — run

   make evaluate RUN_ID=<run_id>
   

   per imported run (the plan lives in .research/evaluation.json; make list-tasks shows what each task type returns). On contract errors, make validate-inputs RUN_ID=<run_id> pinpoints them without scoring. The report includes metrics, curves, skipped (uncomputable metrics with reasons — report these, never fill them in yourself) and provenance. Commit the evaluation outputs.

2. Read the results: fetch_experiment_results with the clone as local_path (after import_run_outputs and a pull) — per run its metrics, evaluation report, figures and provenance entry.

3. Author the analysis yourself, against the hypothesis and the design in the record: for each claim, what its metric shows and whether the preregistered criterion is met, then what the numbers show beyond the claims — including when they do not show what was hoped. Read the experiment code in the clone before explaining a result. Write it in Japanese. The analysis is evidence, not advocacy. get_prompts(step="experiment_analysis", local_path=...) renders the guide with each claim's criterion, predicted interval, observed difference and verdict filled in from the clone; with a model at hand, analyze_experiment(local_path, model) has the model answer the same prompt. Either way the result is a draft to check against the numbers, not the analysis itself.

4. Result charts: build a Vega-Lite spec and render_chart it (pass the clone as local_path) to .research/results/chart/<name>.png — PNG, not PDF. Data numbers must be metric references ("metric:run_1.accuracy"), never literals: the tool resolves them from .research/results/ itself, so a plotted point cannot be invented. The tool appends the chart's spec to .research/record.json as its declaration and commits both in the same step — verification re-renders every chart from its declared spec and fails on differences or undeclared chart files. Bars, areas and rects are drawn from zero: when a scale.domain excludes zero the tool declares them with clip: true (so what is verified is what was drawn), but look at the PNG before citing it — labels, ordering and axis ranges are yours. A chart that needs changing gets a new path, or a superseding declaration via append_to_record and a re-render; the old declaration stays. Render charts before update_record, or re-run update_record afterwards: the chart declaration is a new record commit, and values.tex must be rendered against the latest one.

5. Method diagrams: write text notation (mermaid / graphviz / d2) and render_diagram to .research/results/diagram/<name>.pdf.

6. Commit and push. Reference figures in LaTeX as images/<path> with the full relative path you were given — two runs can each produce accuracy.pdf, and only the full path resolves.

Output: an analysis written into the clone, and committed, verifiable figures.