release-parity-test

Release Parity Test

A repeatable end-to-end check that what the MCP reports matches what Copilot's own web app shows, in both read modes, and that every write is visible to the next read. First run: 2026-10-14, before v3.0 (issues #820–#825). The plan was then reviewed against the code and the docs/bugs/ corpus, which added most of the not run rows.

The case list lives in test-matrix.md. This file is the procedure. To extend the test, add rows there (its "How to extend" section); change this file only when the procedure itself changes.

A full run is long. When time is short, run every [P1] row plus the rows whose Last run is a FAIL, and record the rest as SKIP (not run this time).

When to use

• Before a release (the version-bump PR), after the last feature merge
• After a change to src/tools/live/, src/core/live-database.ts, the write tools in src/tools/tools.ts, the field-selection presets, or skills/ (then include the SKL section)
• The user asks to "test against the web app", "run the parity test", or "check the MCP matches Copilot"

Do NOT use if

• The question is one enum or field against production → bun run smoke
• The user wants a financial analysis → /finance-pulse, /finance-cleanup
• There is no Manual account and the user has not agreed to the write cases being skipped (see Guardrail 2)

Guardrails

1. The web app is ground truth. When the MCP and the web app disagree, the bug is in our code or our description until shown otherwise. Copilot-side quirks get documented, not "fixed".
2. Writes go only to the Manual account the user names in chat for this run. Every created object's name starts with ZZ MCP Test <RUN>, where <RUN> is the run's YYYYMMDD-HHMM, so leftovers from an earlier run cannot be mistaken for this run's objects. The cleanup ledger is a section of the local report file. Log each write before sending it (planned name, account, date, amount, expected balance effect, status sent); when the response arrives, add the id and set created; save after every line. Before any update, split or delete, the id must be in this run's ledger, its account_id must be the Manual account, and a live read must show its name still carries the run prefix.
3. Changing a setting on a real account (hiding it, for example) needs the user's explicit yes in chat for that run, and must be undone in the same run. Approval from a previous run does not carry over.
4. Never delete a split parent. Delete the children only (#822). Deleting the parent double-credits the account balance.
5. PII. Real balances, amounts, account, merchant and place names go only into the local report under docs/superpowers/ (gitignored). test-matrix.md, issues and PRs get PASS/FAIL, ratios and synthetic numbers. The session transcript holds real figures too (tool output, page text): never paste from it into an issue or PR without rewriting the figures.
6. Cleanup always runs, including after a failure partway through. It works from the ledger file, so a new session can finish it: Phase 0 reads the latest RELEASE-PARITY-TEST-*.md and completes any ledger line not marked confirmed-gone before starting. Check the balance after every cleanup group (CL-07). Stop and tell the user, without trying to repair anything, if the balance moves by an amount the ledger does not explain, or if any write reached a non-Manual account.
7. Web automation. The web column needs the Claude in Chrome extension; its skill is provided by the host, not this repo. Load its tools first. If it is unavailable, every web comparison is SKIP (no browser), never PASS on two sources: the web is the ground truth. Read pages with the extension's page-text tool rather than screenshots, and read figures from the page's own search, filters and totals, never by counting rendered rows (lists load more as you scroll). Record the raw text you read. Before filing a FAIL that rests on a web figure, reload and read it again; right after a write, wait 30 seconds and reload once. If an input misbehaves (masked or content-editable fields), stop and ask the user to do that one step; never commit a value you have not read back.
8. A write that errors may still have landed. On any write error do not retry. That includes writeMayHaveApplied, a bulk edit's "N/M succeeded", deleted: false, and a shape error after the call. First run refresh_cache { scope: "transactions", months: [...] }, search live for the run prefix, and check the web. Set the ledger line to what actually happened. Retry only once the object is confirmed absent.
9. Keep test rows away from real data unless the case needs it. When a case needs a real category (TXN-05, a split child), note that category's month total in the web before the write and check it at the end (CL-08). Test names must contain no real merchant name, since a real recurring's rule could pick the row up (REC-05). Use set_budget only on the test category, and restore a budget to its previous value: "0" is a budget of zero, not "no budget".