Assay

Assay — Assertion-First AI Evaluation

Contents: Ideal State · Intent Routing · The taxonomy · Choosing a check: tier first, then shape · Selection, acceptance, and unavailable evidence · The canonical path · Doctrine (from Anthropic; encode, don't restate) · The image lane (visual evals) · Harness integration · Judgment-layer question systems · Data & Config · Gotchas · Examples

An eval gives an AI an input, then applies assertions to its output to measure success (Anthropic's definition). A case is {id, prompt, assert:[...]}. Each assertion is either deterministic (code, free) or model-graded (a calibrated judge). Cases run multiple trials and report pass^k (every trial passes; the reliability metric and the regression gate) and pass@k (any trial passes). Proof applies the same tiers to one run's output (the film, image, page, email or document it made) against what was asked, before the principal sees it: Intents/Proof.md, with the whole system in LIFEOS/DOCUMENTATION/Assay/AssaySystem.md. CreateQuestions designs the fast tier itself: it turns a fuzzy decision into a measured judgment-layer question system (LIFEOS/DOCUMENTATION/Assay/AssaySystem.md § Judgment layer).

Sources: Demystifying evals for AI agents (2026-01), A Statistical Approach to Model Evaluations (2024-11), develop tests, the eval-awareness contamination post (2026-03), and the live claude plugin eval surface.

Freshness contract: "aligned to Anthropic's doctrine" is a live claim. When designing a new suite class or touching ## Doctrine, re-fetch the demystifying-evals post and flag where it has moved. Report divergence, never auto-adopt; an unreachable URL never blocks a run.

Ideal State

• Every check sits at the lowest tier that can honestly answer it (code, glance, judge, human), and only code runs on a blocking path.
• Every score ships with pass^k, SEM and a 95% CI from the trials that completed, with the requested and completed trial counts beside it; fewer completed trials than requested never claims pass^k.
• A judged check is calibrated against the principal's labels or reads PROVISIONAL wherever it shows, and a judged suite keeps an adversarial fixture its judge must fail.
• A produced artifact is checked by Proof against what was asked before the principal sees it, and the receipt is bound to its bytes.
• Transport failures, malformed verdicts and missing observations are reported as unavailable evidence, never as a pass, a quality failure or an adversarial catch.
• A fuzzy decision handed to CreateQuestions leaves as a validated question system whose accuracy was measured on held-out cases the designer never saw, with its margin over one holistic baseline question, or as a stated finding that the judgment layer does not fit and what does; a design ends at a saved shadow registry row and never moves a caller toward enforce.
• When Assay builds the eval system for a project, the project's ISA ends current: every criterion it checks names its suite or glance question in Test Strategy, and the hand-rolled eval code, rows and stale criteria it replaced are removed in the same change.