ai-output-review

AI Output Review

1 · Purpose

Make "the agent wrote it" pass the same bar as "a junior hire wrote it", read it, run it, and check it against the spec, because AI output fails in different places than human output.

2 · When to use

• Before accepting any non-trivial agent-produced diff, document, or config.
• When reviewing a teammate's PR that smells AI-generated and unreviewed.

3 · Inputs

• The spec or task statement the agent worked from.
• The diff or artifact.
• The project's verifiers: tests, linters, type checks, evals.

4 · Procedure

1. Read the diff against the spec first. Ask of every changed block: which spec line does this serve? Unmotivated changes are the AI smell parade's grand marshal.
2. Run the verifiers yourself. Tests, types, lint, on your machine or CI, never from the agent's claim that they passed. The agent's "all green" is a statement about its context window, not about the repo.
3. Scan for secrets and egress. New endpoints, new domains, tokens in comments, telemetry you did not ask for. AI loves a helpful analytics call.
4. Run the smell list. Each is innocent alone; three in one diff is a rewrite:
   • Unused imports, variables, and "future-proofing" abstractions.
   • Comments that narrate the code instead of the why.
   • Error handling that swallows (except: pass, empty catch).
   • Dependencies added for one-liners.
   • Config keys that nothing reads.
   • Docstrings describing a different function than the one below them.
5. Check the blast radius. What runs this code, with which credentials, on what data? Match the review depth to that answer, not to the diff's size.
6. Require the agent's own evidence. If the workflow produced a trace, test output, or eval score, it must be attached. No evidence, no merge.
7. Decide: accept, request changes (specific, numbered), or reject. "Looks fine" after 30 seconds on 400 lines is a rejection of the review, not an acceptance of the code.