eval

Skill: Eval (live gold-suite runner)

Purpose

Run the analyst on every question in the held-out gold suite, then score the answers against the blind gold: accuracy (the analyst's number vs the recomputed gold), query-similarity (its SQL vs the approved query), and cost/latency. This is the system-level eval — the climb the Context pillar moves and the number the model comparison turns on.

Two honest properties:

• Blind by construction. The analyst runs see the question only — never the gold sql or value. The gold is read only at grading, after the answers are locked.
• Real, not staged. Each answer is produced by actually running the analyst now. Nothing is pre-filled.

Invocation

/eval [train|test|all] [--slice N] — default split train.

• train — the working set you iterate on (error-analyze, add context, watch it climb). Default.
• test — the held-out set. Run this ONCE at the end as the honest generalization number. Never iterate against it (D8).
• --slice N — run only the first N cases (the in-room live slice). Omit for the full split.

Examples: /eval train · /eval train --slice 3 · /eval test

How to run it

Step 0 — preflight (D3, fail loud)

from helpers.eval_driver import preflight
conn = preflight()   # raises clearly if Snowflake isn't live — there is NO DuckDB fallback

If it raises, stop and surface the message. Do not grade against any other engine.

Step 1 — get the questions (blind)

Load the question set for the split. This returns questions only — no answers — so you cannot leak the key:

import sys; sys.path.insert(0, "<ai-analytics-evals path>")
from aievals.data.gold import load_questions
questions = load_questions("<...>/aievals/data/novamart_gold.yaml", split="train")  # [{question, split}]

If --slice N was given, take the first N.

Step 2 — run the analyst once per question

Launch one fresh sub-agent per question with the Task/Agent tool (run them concurrently in reasonable batches). Each sub-agent gets a fresh context and sees ONLY its question. Time each run (wall-clock) for latency. Give each sub-agent exactly this brief: