verify-cascade

Verify-Cascade — extract cheap, verify, escalate only on a flag

maggy already routes by task complexity. This makes routing output-aware: a cheap model does the work, a decomposed verifier checks it, and you escalate to a strong model (Claude) only when the verifier flags a real problem. The cheap rung handles the easy items for near-free; only flagged items pay for the strong model — most of the quality of the big model at a fraction of the cost.

It's the same discipline as the security-audit finder ≠ validator gate, applied to routing: an independent verifier catches the cheap model's blind spots.

The loop

1. EXTRACT   cheap model produces an output (extraction / answer / summary)
2. VERIFY    a decomposed verifier asks narrow per-field yes/no questions,
             framed so "bad = true", and returns P(wrong) per question
3. GATE      escalate if ANY question's P(wrong) > fire threshold (max, not mean)
4. ESCALATE  only the flagged items pay for the strong model; the rest keep the cheap answer

Schema-validation is necessary but not sufficient: a cheap model produces confident, schema-valid fabrications (a blank field filled with a plausible invented value). A structural check can't see that — a semantic verifier can.

Verifiers (pluggable) — verify_cascade.py

VC="$(cat ~/.claude/.bootstrap-dir)/skills/verify-cascade/verify_cascade.py"
python3 "$VC"          # self-test (stubbed judge, no network, no model)

LocalVerifier — the default, nothing leaves your machine

Runs the whole decomposed question battery through a cheap CLI model in one call and returns {field::metric: P(wrong)}. No external service.

import sys
sys.path.insert(0, f"{__import__('subprocess').check_output(['cat', __import__('os').path.expanduser('~/.claude/.bootstrap-dir')]).decode().strip()}/skills/verify-cascade")
# (or simply: sys.path.insert(0, "<bootstrap-dir>/skills/verify-cascade"))
from verify_cascade import get_verifier, build_questions, cascade
v = get_verifier("local")                     # uses $MAGGY_JUDGE_CMD (default: deepseek --flash)
result = cascade(
    extract  = lambda: cheap_extract(...),     # your cheap-model call -> dict
    verify   = lambda rec: v.verify({"source_text": source}, build_questions(rec, schema)),
    escalate = lambda: strong_extract(...),    # your strong-model call -> dict
    fire_t   = 0.7,
)
# result: {record, escalated: bool, fired: [qid...], scores: {qid: p}}

Set the judge model with MAGGY_JUDGE_CMD (e.g. qwen3, deepseek --flash) or pass your own judge=callable(prompt)->text. The default passes the prompt on stdin (no shell, so prompt content is never interpreted as a command), so MAGGY_JUDGE_CMD must read from stdin. Questions are scored in complete batches and an over-long source warns on stderr rather than being silently truncated.