octocode-graph-eval

Octocode Graph Eval

Evaluate outcomes and run improvement loops with evidence, not vibes — for one loop or a graph of loops. Flow: ERROR-ANALYZE → FRAME(goal→KPI) → BASELINE → LOOP → JUDGE → CAPTURE → VERIFY → SUITE-EVOLVE. Modes: ErrorAnalyze · Define · Run · Suite · Benchmark · Audit.

Lobby rules

• No goal→KPI link → STOP. No measurable primary → STOP. No runnable sensor → build one before looping.
• Narrative-only accept → REJECT. Editing harness/cases/graders to pass → REJECT.
• ACCEPT only if primary moves on held-out and guardrails hold.
• Prefer deterministic graders; binary/LLM next; humans calibrate. Grade outcomes over paths.
• TDD for agents: write or select a failing case / KPI check before mutating the subject; green only after the change (red → green → keep|discard).
• Public benches orient; private failure suites gate ships. Distrust saturated/contaminated boards.
• Freeze the harness during an experiment; evolve the suite only between experiments.
• Graph check: before evaluating a multi-agent workflow, run edge detection — if no two nodes are independent (every step reads the prior step's output), it is a loop, not a graph. Build a loop.
• Goodhart guard: every primary KPI must have a counter-metric guardrail the agent cannot tune. Primary improving + guardrail degrading → reframe the goal, not the loop.
• Verifier independence: a verifier sharing the executor's context is not independent. Require fresh context before calling a result verified.
• Anchor requirement: every graph must have at least one node whose output cannot be argued with (tests that ran, build exit codes, type errors). No anchors → build one before trusting the graph.

Workflow

1. Error-analyze traces into a failure taxonomy; frame success, primary/leading metrics, guardrails, and decision rule.
2. Measure a fixed-budget baseline; make the smallest subject change; keep or discard from comparable results.
3. Judge grader quality, fairness, capability versus regression, and contamination; capture one durable lesson.
4. Verify held-out results and required checks; then add new failure cases between experiments. Stop when goal/KPI is undefined, checks did not run, the harness changed to pass, or another loop cannot change the verdict.