wandb-track-experiment

Track Experiments with W&B

Instrument an experiment loop so every candidate becomes a comparable W&B run with enough context to understand what changed, what happened, and whether the objective improved.

Operating Rules

• Treat W&B as an observation layer. The experiment loop, including a Ralph loop, must continue making decisions from local results rather than a required round trip to W&B.
• Use one W&B run per candidate experiment, not one long run for the entire research session.
• Use a shared run group to connect iterations from the same experiment session.
• Keep config for independent variables and provenance, run.log() for changing measurements, and run.summary for final comparison values.
• Preserve the existing experiment command, objective calculation, retry behavior, and stopping logic.
• Prefer structured IDs and short summaries over full prompts, source files, or large generated outputs.
• Keep live alerts disabled until Scriptable run alerts and a destination are configured.
• Verify exact SDK and alert behavior against current official W&B documentation before implementing brittle details.
• Use W&B Models for experiments and system metrics; use weave-add-tracing for model, agent, and tool traces.

Workflow

1. Find the Research Loop Contract

Identify:

1. The experiment loop controller and candidate entrypoint.
2. The unit that represents one iteration or candidate.
3. The primary objective name, value, and maximize or minimize direction.
4. Baseline value or baseline run, if one exists.
5. Iteration, time, cost, and retry budgets.
6. Where hypothesis, code change, seed, dataset, command, and result are already represented.

Read references/experiment-tracking-patterns.md before changing code.

2. Define the Run Contract

Default lifecycle:

Experiment session
|- candidate iteration 0 -> W&B run
|- candidate iteration 1 -> W&B run
`- candidate iteration 2 -> W&B run

Use these W&B fields consistently: