ralph-experiment

[EXPERIMENT LOOP — ITERATION {{ITERATION}}/{{MAX}}]

<Use_When>

• User wants to iteratively optimize something (performance, accuracy, reliability)
• User says "experiment", "optimize", "improve performance", "try hypotheses", "karpathy loop"
• The task has a measurable success criterion (latency < X, throughput > Y, error rate < Z)
• Multiple approaches are possible and the best one isn't obvious
• User wants to explore a solution space methodically </Use_When>

<Do_Not_Use_When>

• Task has a clear implementation path -- use ralph instead
• User wants to implement a known feature -- use ultrawork or ralph
• There's no measurable metric to optimize -- use ralph with acceptance criteria
• User wants a one-shot fix -- delegate directly to an executor agent </Do_Not_Use_When>

<Why_This_Exists> Optimization tasks fail when treated as implementation tasks. "Make X faster" doesn't have a single correct solution -- it requires forming hypotheses, testing them, learning from failures, and iterating. Ralph-experiment provides:

1. Structured hypothesis tracking with predictions and outcomes
2. Git checkpoint/revert so failed experiments don't pollute the codebase
3. A queryable experiment notebook that captures learnings across iterations
4. Flexible termination: success criteria, budget limits, or manual interrupt
5. Agent/team delegation to increase hypothesis success rate
6. A simplicity criterion: prefer fewer lines of code at equal performance </Why_This_Exists>

<Execution_Policy>

• One hypothesis at a time -- sequential experiments, not parallel
• Delegate implementation to executor agents or teams for each hypothesis
• Use run_in_background: true for measurement commands (builds, benchmarks, test suites)
• Always pass the model parameter explicitly when delegating to agents
• Git commit before measuring, git reset on discard -- no uncommitted experiments
• Never stop unless: success criteria met, budget exhausted, or human interrupts </Execution_Policy>

Step 1: SETUP (first iteration only)

Parse the user's request to extract:

• Goal: What are we optimizing? (e.g., "Reduce p95 latency for BatchAck messages")
• Success criteria: Measurable targets (e.g., p95_latency_ms < 120)
• Measurement command: How to measure (e.g., npm run benchmark -- --type=BatchAck)
• Budget: Termination limit. Parse from args:
  • --budget N → N experiments max
  • --budget Nh or --budget Nm → time limit
  • --budget "scripts done" → scope-based (scripts complete successfully)
  • Default: 20 experiments

Then: a. Create experiment branch: git checkout -b experiment/{tag} from current HEAD b. Initialize experiment-notebook.json in .omg/ (see schema below) c. Run the measurement command to establish baseline metrics d. Record baseline in the notebook