eval-engine

Eval Engine

This skill powers AI Evals for PMs. Its stable install and CLI identifier is pm-verifier.

Keep the user-facing flow simple:

product contract → define surfaces → create eval suite → run trials → grade → inspect failures → release decision

Put framework complexity in the bundled harness, not in the PM's workflow.

1. Start from the claim

Read the spec or existing suite. State the decision the evaluation must support:

• Capability: what can the candidate do, and how often does at least one attempt succeed?
• Regression: can the candidate still perform previously reliable behavior consistently?
• Safeguard: does a safety or privacy control block forbidden behavior?

Ask for missing product facts. Never invent policies, expected fields, thresholds, reference answers, or safety boundaries.

Classify each approved claim against four surfaces:

• Outcome: the final user/environment state.
• Trajectory: the risk-critical path, tools, policies, and decisions.
• System: required and optional checkpoints, identity, continuity, first failure, and consequences.
• Memory: only when the product promises persistence; write, retrieve, update, forget, isolation, freshness, conflict, and time semantics.

Safety, privacy, reliability, quality, and operations are cross-cutting grader categories. Capability and regression are suite lifecycle purposes.

2. Define good

Separate:

• Gates: binary, disqualifying checks on one trial. Prefer deterministic code checks for objective facts, outcome state, tool calls, required structure, safety, and privacy.
• Rubric criteria: gradual 1–5 judgments for useful differences among gate-clearing outputs. Use feature-specific anchors and a worked example.

Use the smallest complete set. Do not add weak gates to satisfy a fixed count. Read references/gate-design.md when a check could be either a gate or a score.

3. Create the suite

Create the files in references/evidence-contract.md. Record: