agentsop-test-fix-loop

Test-Fix Loop · SOP

One-liner: The test result IS the next prompt. Wiring the verifier is 20% of the work; framing its output as a useful feedback message is 80%.

---

1. 何时激活 (Activation Rules)

Activate this skill when any of the following triggers fire:

• The user says "have the agent fix until tests pass", "run lint and tests automatically", "iterate until green", or invokes aider --auto-test, cline --yes, or an OpenHands-style headless agent.
• The task has a verifiable success command: a non-zero exit code on failure (pytest, ruff, mypy, eslint, tsc, go test, cargo check, npm run build, make check, …).
• You're wrapping a code-editing LLM in a script/CI step and need to decide: when does the agent return?
• The agent just made an edit and the next message in the loop would be "here's what the verifier said".

Do not activate when:

• Success is subjective (writing prose, designing UX). The loop has no feedback signal worth replaying.
• The verifier is slow + interactive (full E2E suite, multi-min builds). Either async-ify the loop, or run a fast subset (pytest -x -k changed) in the loop and gate the slow suite at PR review.
• The gate is human approval, not a machine check — use the HITL skill.

---

2. 核心心智模型 (Core Mental Model)

2.1 The test result IS the next prompt

The agent's next turn is conditioned almost entirely on the message you inject between edit-N and edit-N+1. That message — formatted from stdout, stderr, exit_code — is the prompt. The framework labels it "tool result" or "verifier output" but mechanically it is a user-role message the LM consumes verbatim.

⇒ Framing the feedback dominates the model choice. A 4000-line raw pytest dump prompts a worse fix than a 30-line "first failing test, traceback, the diff you just applied" digest, regardless of the model behind it.

2.2 Four primitives

+-----------------+   +-----------------+   +-----------------+   +-----------------+
| 1. Verifier     |   | 2. Capture      |   | 3. Format       |   | 4. Iteration    |
|    command      |   |    (stdout +    |   |    feedback     |   |    bound        |
|                 |   |     stderr +    |   |    message      |   |                 |
| - pytest -x     |   |     exit_code)  |   | - first error   |   | - max N tries   |
| - ruff check    |   | - timeout cap   |   | - last K lines  |   | - escalate /    |
| - mypy --strict |   | - byte cap      |   | - drop noise    |   |   commit / skip |
| - eslint .      |   | - kill on hang  |   | - keep colors=0 |   |                 |
+-----------------+   +-----------------+   +-----------------+   +-----------------+