exp-pilot-eval

/exp-pilot-eval

Evaluate pilot experiment results and update the linked idea page. Reads pilot results from experiments/pilot/code/{slug}/results/, applies verdict logic (pass/fail/inconclusive), and updates the idea's pilot_result, failure_reason, and status fields. Called by /ideate Phase 5 after /exp-pilot-run completes.

Inputs

• idea-slug: slug of the idea whose pilot was just run
• --auto (optional): automatic mode, no pause for user confirmation (used when called by /research)

Outputs

• wiki/ideas/{slug}.md — updated pilot_result (always), failure_reason (if failed), status (if failed → failed)
• wiki/log.md — appended log entry
• PILOT_VERDICT_REPORT (printed to terminal) — verdict, wiki change summary, next step suggestions
• experiments/pilot/{slug}/report.md — PILOT_VERDICT_REPORT file copy (persistent record)

Wiki Interaction

Reads

• wiki/ideas/{slug}.md — linked idea current state: status, pilot_result, failure_reason
• experiments/pilot/code/{slug}/results/seed_*.json — pilot experiment results
• experiments/pilot/code/{slug}/pilot.log — pilot run log (for failure diagnosis: errors, warnings, runtime behavior)
• experiments/pilot/{slug}.yaml — Pilot Spec (for success_criterion and baseline reference)

Writes

• wiki/ideas/{slug}.md — update pilot_result, failure_reason (if failed), status (if failed)
• wiki/log.md — append operation log
• experiments/pilot/{slug}/report.md — PILOT_VERDICT_REPORT file copy

Graph edges created

• None. Pilot evaluations do not create graph edges (those are for formal experiments via /exp-eval).

Workflow

Step 1: Load Context

1. Read idea page wiki/ideas/{slug}.md:

   • Current status and pilot_result
   • Current failure_reason (should be empty for proposed ideas)

2. Read pilot results from experiments/pilot/code/{slug}/results/seed_*.json:

   • Parse result files (JSON)
   • Compute mean ± std per metric (across seeds)