write-experiment-code

Write the experiment code

Needs a clone with its research context committed and the execution platform settled — the platform's reference under _shared/references/ states the architecture and environment constraints the code must satisfy, so read it before writing.

However the code is produced — authored here or by an external code-generation tool — the contract below is what the repository holds it to; swapping the producer changes nothing else.

1. The execution contract. The experiment repository ships empty source files; this is what they must become. Run ids and output metric paths must match .research/record.json exactly — verification rejects results directories no declared run accounts for, and a claim whose metric the code never emits can never be realized. Library docs via get_library_docs.

   Edit or create ONLY these files (.github/ is managed by AIRAS, and everything must run on a Linux runner):

   Path	Role
   Dockerfile	Reproducible environment (Python 3.11 + uv)
   config/config.yaml	Shared Hydra defaults
   config/run/*.yaml	One run config per (method, model, dataset)
   src/main.py	Orchestrator for a single run_id (Hydra entrypoint)
   src/preprocess.py	Dataset loading / preprocessing
   src/train.py / src/inference.py	Single-run executor
   src/model.py	Model definition, if a custom one is needed
   src/evaluate.py	Independent aggregation script
   pyproject.toml	Dependencies only

   The CLI shape is fixed — run_experiment.yml and the external executors call exactly this, so it cannot change:

   uv run python -u -m src.main run={run_id} results_dir=.research/results mode={sanity|pilot|full}
   uv run python -u -m src.evaluate results_dir=.research/results run_ids='["run-1","run-2"]'
   

   Run ids are {method_type}-{model}-{dataset}, dropping whichever of model/dataset does not apply; method_type is proposed or comparative-{index}.

   All three modes must work, on the same dataset and model — only the scale changes: sanity cheap enough to run locally on CPU (1 epoch, 1–2 batches, or 5–10 inference samples), pilot 20–30% of full (≥3 epochs, ≥50 samples) for a go/no-go, full the real thing. sanity and pilot log to {project}-sanity / {project}-pilot so they never pollute the full runs.

   sanity prints SANITY_VALIDATION: PASS with a SANITY_VALIDATION_SUMMARY: {...} line, or SANITY_VALIDATION: FAIL reason=<short_reason>; pilot prints the PILOT_VALIDATION equivalents. Checks, adapted to the task: ≥5 steps with final loss ≤ initial, or ≥5 non-identical outputs; every metric finite; FAIL reason=missing_metrics when they are absent. This is your own gate before dispatching anything expensive — nothing downstream parses it for you.

2. Write the three files verification reads. Per run, under {results_dir}/{run_id}/:

   File	Written by	Why it is required
   eval_inputs/<task>.json	src/main.py	the raw predictions; what the metrics can be re-derived from
   evaluation/<task>.json	make evaluate	airas-eval's verdict, its versions and skipped
   metrics.json	src/evaluate.py	copied from the airas-eval report; the file the record is checked against

   Do not write out the resolved configuration. The record takes the parameters a run executed with from the platform's record of the dispatch, never from a file the run wrote: code reporting its own settings can report anything, so agreeing with it would prove only that the code was willing to claim it.

   All three go under .research/results/: that is the only tree the executor collects back, so anything written beside the Hydra logs or in a scratch directory never reaches the repository, and the check that would have used it silently passes on an empty value.