proof-looping

proof-looping — "Make no mistake, but for real this time"

No "done" without proof — and the proof is scored by an independent judge, never a deterministic heuristic alone. (In the reference run, a deterministic check false-PASSED an auth-gated task twice, fooled by the app's own demo/template content; only an independent visual judge + a content-match to the task caught it. Preventing that exact lie is why this skill exists.)

Covers six keywords: agent harness · benchmark · UI · prod agent · loop engineering · agentic RL.

The loop (8 phases)

1. Set up proofloop — drop surface-bench.mjs (UI breadth) + proofloop-run.ts (harness depth), add proofloop:ui / proofloop:engine npm scripts, an AGENTS.md completion gate, and a CI job. Enforcement lives in CI/branch-protection, NOT in the agent's good intentions.
2. Ingest codebase + agent harness — map the REAL runtime: the loop (classify → plan → tool → execute → synthesize), the model seam (how tool/model calls are injected — e.g. a callTool("call_llm") callback), routes, and any existing eval infra. Ground in real files (read them); never assert a file/flag you didn't open.
3. Intake UI + its mental model — enumerate every surface (router paths + the agent-readable screen registry, e.g. data-screen-id), the user's job on each, and the interactive affordances (composer input, submit, completion signal). One dominant job per surface.
4. Research design references — pull the top web-design exemplars for the app's category; turn them into the visual-judge rubric (first-pixel-is-action, one job per screen, no overflow, visible loading/empty/error, screenshot-worthy output).
5. Research benchmark — find the benchmark matching the deliverable shape, not the hype: spreadsheet edits → SpreadsheetBench; coding repair → SWE-bench; tool-use → BankerToolBench; research-with-sources → GAIA / FRAMES / SimpleQA or the app's own persona/tri-search evals. Picking a benchmark for the wrong shape is the most common waste.
6. Pick benchmark — choose it, then write the task set + per-task acceptance criteria (a concrete expect per surface/task — a blanket "page has >200 chars" rule will mis-grade a graceful 404 and pass a blank demo). Keep a held-out split; no answer-keys.
7. Set up LOCAL env with benchmark tasks — run the app locally so YOU control auth, seed data, and model. This is deliberate: prod auth is often OAuth = a real wall, and benchmarking prod pollutes it. Local lets every task run repeatably, browser-verified, with a seeded/test session and the model you choose (point the harness's model seam at your target model for parity).
8. Run it · ship it · prod-proof it · loop it — proofloop:ui drives every surface in a real browser (screenshot + video + console + deterministic UI-contract checks + visual judge + interactive submit→result task); proofloop:engine runs the real harness on your model and exports an (s,a,o,r) trace. Gate: a surface counts only if render PASS && task PASS && visual ≥ 1. Promote every failure to a regression check. Loop until the gate is green; the traces become agentic-RL reward data.