golden-bite

Golden bite

An AI step matches incoming bank transfers to open invoices, or returns UNMATCHED for a person. The golden set has 240 cases written at launch (71 of them agent-written rewordings), and the feared harm (a payment booked to another customer's invoice) has its own zero-miss line. Prompt v7 adds "if the remittance contains an open invoice number, match it". It scores 240/240 and ships. Nine business days later, 37 payments are on the wrong customer: "PO 10452" was the payer's own purchase order, and INV-10452 belonged to someone else. Replayed on v7, the golden set still scores 240/240. In every case the invoice number, payer and amount agreed, so any version got them right.

Covered is not caught. Prove a case can fail before you trust it to pass.

Part A — Ask (project rules, or paste after a bug ticket for an AI step)

When you fix a mistake an AI step made:
1. MISTAKE: the real input, exactly as it came in, and what the step did with it.
2. CASE: add it to the golden set with expected left blank. I'll get it from the owner.
3. BITE: run the new case on the version that made the mistake. It must fail.
   If it passes, say TOOTHLESS and rewrite it from the real input.
4. Only then change the prompt or model. Report: new cases | fail on old | pass on new.
Never edit an expected label. If one looks wrong, reply:
LABEL DISPUTE | case | why, and stop.

In an orchestrated flow (spec → plan → tasks → implement): name who labels each AI step's golden cases in the spec; when a task fixes an AI mistake, its first step adds the mistake as a case and shows it failing on the current version; each task review shows the bite count and any label changes with the owner's name.

Part B — Check (you do this)

bite_check.py has no dependencies. It reads a golden file (id,source,labelled_by,expected) plus results files (id,predicted) from one or more known-bad versions and a candidate. It counts, per source, how many cases fail on a known-bad version (their teeth) and how many the candidate passes. Cases labelled by the agent, or by nobody, aren't counted. Replay the story on made-up files: