arch-eval-catalog

Eval catalog

You cannot optimize what you cannot re-test. An eval set lets a developer cut a model call or shorten a prompt and know the same day that nothing broke. Coverage means conversation scenarios with evidence, not a test count.

Three layers of testing

1. Offline, before deploying: keep the exported package in version control and run debug_lint_abl and platform_validate_package on it; debug_why_transcript_failed maps a bad transcript to ABL file and line. Point tools at mock backends when the real ones are not ready, so tests do not wait on another team.

2. Live scripted conversations: debug_load_agent and debug_send_message play multi-turn conversations against the working copy; check debug_get_current_state after each turn.

3. Platform evals: platform_eval_personas, platform_eval_scenarios, platform_eval_evaluators, platform_eval_sets and platform_eval_runs. Target the working_copy, an environment (dev, staging or production), a deployment or an exact agent version. Use preflight before a big run, quick for a smoke check, compare between runs and heatmap to see which scenarios regress.

4. Repair loops: platform_arch_auto_loop runs governed repair of failing scenarios. Read each proposal and the gate results that accepted or blocked it (list_proposals) before applying, and re-run the eval set afterwards.

Build the catalog

1. Read the experience contract (arch-experience-contract) for the top intents and success definitions.
2. Draft with platform_eval_personas and platform_eval_scenarios using generate across every generation category (happy path, edge case, error handling, multi-turn, handoff, adversarial). Review before accepting; generated catalogs miss hard cases.
3. Cover every family in references/scenario-families.md, and include the adversarial persona templates (prompt injector, social engineer, off-topic, abusive, edge-case explorer).
4. Make every scenario checkable: an expectedOutcome, maxTurns, and at least one deterministic expectation (expectedMilestones, agentPath, requiredToolOrder or maxToolCalls). Use responseOutcomeContract or requiredContextFieldContracts for values that must be right, and toolFixtures to make tool outcomes repeatable.
5. Set turnLatencyBudgetMs yourself where latency matters; only an explicit budget is asserted.
6. Turn every diagnosed production failure into a scenario before fixing it.
7. Run each scenario at least three times (variants) and read the spread, not just the average.