convex-verify

Prove a feature works — seed, drive, assert

A green typecheck proves the code parses; it does not prove a non-owner is actually denied, that a query returns the right rows, or that a mutation has the effect it claims. This capability closes that gap with the loop the whole field is missing: seed → drive → assert, run in-process with convex-test so it needs no deployment. Its highest-value assertions are the NEGATIVE ones — the caller who should be refused — because those are exactly the authz defects the 30-app corpus shows are the #1 real bug and the ones a happy-path demo never catches.

Steps

1. IDENTIFY the feature to prove: the specific exported query/mutation/action (or a small set) the user just built/changed, and its intended behavior — who should be allowed, what data should come back, what a mutation should change. If the intent is unstated, ask one focused question rather than guessing the contract.
2. SET UP convex-test: ensure convex-test + vitest are dev deps AND a vitest.config.ts sets test.environment: "edge-runtime" with server.deps.inline: ["convex-test"] — WITHOUT that config, convexTest(schema) fails at runtime with import.meta.glob is not a function (verified). Also install @edge-runtime/vm. Then convexTest(schema) gives a t handle. Reuse the project's existing test setup if present (compose with the test capability, don't fork it).
3. SEED realistic data through the app's OWN functions where possible (so the seed exercises the same validators/mutations a real user would), falling back to t.run(async (ctx) => ctx.db.insert(...)) for fixtures the public API can't create. Seed at least: the caller's own rows AND a second user's rows, so cross-user access is testable.
4. DRIVE the feature as DIFFERENT identities with t.withIdentity({ subject, tokenIdentifier, ... }): call the function as (a) the legitimate owner, (b) a different authenticated user, and (c) unauthenticated (t with no identity). Use the real identity shape the app's auth uses (subject/tokenIdentifier), matching how ownership is resolved.
5. ASSERT behavior — POSITIVE and NEGATIVE:
   • positive: the owner gets the expected rows / the mutation made the expected change (expect(await t.withIdentity(owner).query(api.x.y, args)).toEqual(...)).
   • NEGATIVE (the load-bearing half): a different user calling the same function is REFUSED — await expect(t.withIdentity(other).mutation(api.x.cancel, {id})).rejects.toThrow(/forbidden|not authorized|403/) — and an unauthenticated caller is refused where auth is required. A feature is not proven until the wrong caller is shown to be blocked.
   • data-scope: a list/query returns ONLY the caller's rows, never the second user's (assert the second user's row is absent).
6. RUN the tests (npx vitest run) and report: what was proven (each positive + negative assertion that passed), and — critically — any assertion that FAILED, because a failed negative assertion is a real authz hole found before ship. Emit findings on the bus (specs/finding.schema.json, class authz/correctness, evidence kind probe-result with the exact failing call) for anything that didn't behave.
7. Do NOT weaken a test to make it pass: if the owner-only query returns another user's row, the FIX is in the function (hand to convex-authz), not in the assertion. A test changed until it's green proves nothing.