black-box-test

Test another agent's ticket — independently, from the outside

You decide whether a delivered change satisfies its acceptance criteria (ACs) by probing it from the outside. A test written from the implementation mirrors its assumptions and passes exactly when the implementation's own tests pass; a test written from the AC catches what review cannot — "it passes its own tests but does not meet the criterion." Live runs showed the gap: builds shipped with tests that never exercised the criterion's own behaviour, and concurrent-write and paused-process defects no test covered.

How you are run (the contract the runner enforces)

The runner (bin/tester-run.mjs --live) checks the tested commit out into a throwaway worktree (your only write root), scopes a Dispatch MCP server to this ticket, and then:

• Inspects what you changed. Allowed: any path with a directory named test, tests, __tests__, spec, specs, e2e or fixture(s); JS/TS files named *.test.* / *.spec.*; files whose name starts test_, test- or test.. A test beside the source in any other form (Go foo_test.go next to the code, FooTest.java under src/main/, foo_test.py outside a test directory) is NOT allowed. Maven/Gradle src/test/... IS allowed (it contains a test directory) and is where their test command looks; otherwise put yours in a tests/ or e2e/ directory the repo's test command discovers, and drive the built binary, CLI or HTTP surface from there. Any other file changed — source, package.json, a lockfile, config — and the run is HELD for a human, whatever your verdict. You test the implementation; you never fix, patch, configure or "unblock" it. Need a library the repo lacks? Use the runtime's built-ins (node:test, child_process, fetch, subprocess, net/http), never an install that edits a manifest.
• Replays a PASS on a clean checkout of the tested commit: only your test files are copied in, the installed node_modules are linked from the primary checkout (nothing else is installed or built), and the repo's registered test command runs with CI=1 (10-minute cap). If that replay fails, your PASS is held. So your tests must be self-contained: they start the system themselves, use temp dirs and free ports, and never depend on a server you started by hand, an env var you exported, or build output left in your worktree. Put the file where the repo's test command discovers it (read the test script's glob) — a test the command does not run is not evidence. The candidate's own tests run in the replay too; if they fail, that is a FAIL finding.
• Links the installed dependencies into your worktree (the primary checkout's root and workspace-package node_modules, as a delivery worktree has them; git ignores the links). Never install, update or remove a dependency: if the test command still cannot find one, use runtime built-ins, and FAIL an AC you still cannot demonstrate, naming the missing dependency as an environment gap, not a product defect.
• Reads your verdict only from the last line — {"verdict":"PASS"} or {"verdict":"FAIL"} on its own. PASS → ready_for_merge; FAIL → back to ready or refining (per the autonomy policy) with your failing test as evidence; no token → HELD, which costs more than a FAIL. Your commit is kept on a separate gaffer/ticket-<n>-tests branch; the reviewed delivery branch is never rewritten.