replay-app-harness

Replay the user's app on frozen benchmark tasks

Model arms answer "what would model X do?". The app_replay arm answers "what does the user's app, as the code currently exists, do on the same frozen tasks?" — the regression check to run after editing the user's code. Full reference: docs/app-harness.md in this repo; sidecar schema: schemas/understudy.app_harness.v1.schema.json.

Resolve CLI

Prefer the installed understudy binary. If it is unavailable inside a repo checkout, run through the package script:

npm run build
node dist/bin.js runs execute --benchmark <benchmark-dir>

Safety Gates

• The harness launches the user's own code as a subprocess. Only put commands in app-harness.json that the developer showed you or approved; never invent entrypoints. Show the drafted sidecar and get approval before the first run.
• LLM traffic is pinned to the Understudy gateway. The executor's redirect env vars win over the harness's env — do not attempt to route around them, and say so if the app hard-codes a provider base URL.
• The app may mutate real state. Before queueing, confirm with the developer that the entrypoint is replay-safe (test/sandbox mode, no production writes). If unsure, do not run it.
• Honest scoring only. Never present app_replay_unobserved or unscored rows as passes/failures; report them as unobserved and point at the tier-1 boundary in docs/app-harness.md.

Step 1 — Author app-harness.json from the user's repo

Read the user's repo and draft the sidecar into the benchmark directory:

1. Find the entrypoint that handles ONE task-shaped request end to end (CLI/worker script preferred; long-running HTTP servers are tier 2).
2. Study how the task input reaches it and pick input_mode:
   • argv — prompt appended as the final argument;
   • stdin — one JSON line {"task_id", "prompt"};
   • http — author the endpoint template now; it validates but does not execute at tier 1 (say so to the user). In every mode the app can instead read UNDERSTUDY_TASK_PROMPT / UNDERSTUDY_TASK_ID from the env.
3. Note the SDK shape in notes (OpenAI/Anthropic — both base-URL redirect vars are injected; see the instrument skill).
4. Set tool_route: "gateway_tools" only if the app's tool layer writes UNDERSTUDY_LIVE_JOURNAL lines; otherwise "none" and tell the user the rows will honestly read app_replay_unobserved until tier 2.
5. Set per_task_timeout_seconds (default 300, max 3600), cwd, command (argv vector), schema_version: "understudy.app_harness.v1", llm_route: "gateway".