eval-suite-planner

Purpose

This skill produces the Plan artifact of the /eval-guide lifecycle: a populated copy of the customer's Eval Suite Planning & Logging Template plus an interactive HTML review page. The workbook is the source-of-truth artifact; do not replace it with a scenario table, quality-signal table, generic spreadsheet, default .docx report, or HTML-only plan.

The skill aligns to skills/eval-guide/playbook.md and skills/eval-guide/eval-suite-template.md. Use the 10-step playbook as the methodology spine and the XLSX template as the output shape.

Core rule

Copy the blank XLSX template and populate existing cells/rows only. Do not modify the template.

Do not rename sheets, add sheets, delete sheets, add columns, change headers, rewrite README text, edit Dropdown Lists, change styles, change data validation, or convert the template into a different spreadsheet.

If a blank template workbook is available in the session, use it. If not, ask the user to provide the template; do not silently invent a new workbook.

Question policy

Ask targeted questions only when a workbook field materially affects the plan and cannot be inferred safely:

1. Eval owner / named approver.
2. Lifecycle stage and target deployment decision.
3. Whether the agent is prompt-only, RAG/knowledge-grounded, or agentic with tools/connectors.
4. Regulated/compliance obligations.
5. Authoritative sources and source owners.

If the user wants speed or cannot answer, populate TBD - confirm before baseline.

Planning method

When invoked as /eval-suite-planner <agent description>:

1. Extract or infer the agent's purpose, users, knowledge sources, capabilities, boundaries, architecture, lifecycle stage, and known risks.
2. Populate Step 1 — Plan the Eval Effort:
   • one-sentence eval objective;
   • five-factor risk tier: reach, criticality of error, autonomy/blast radius, regulatory/compliance exposure, data sensitivity;
   • one accountable owner.
3. Define eval sets, not scenarios:
   • Capability eval sets: one row per capability dimension that must be diagnostic, e.g. accuracy/correctness, faithfulness/groundedness, relevancy, style/tone, reasoning/tool use.
   • Trust & Safety eval sets: one row per refusal, boundary, or safety category, e.g. guardrails, out-of-scope handling, sensitive-data handling, prompt injection/jailbreak, compliance-specific behavior.
4. Apply Step 4 v5 gates/improvement-target logic:
   • T&S sets use absolute pass-rate hard gates, usually near 100%.
   • Capability sets usually use a launch floor for first deployment plus regression/direction after baseline, not a standing absolute pass-rate target.
   • High-risk capabilities that function like guardrails keep explicit hard floors.
   • Use the template's existing Target pass rate, Target rationale, Gate type, Intended use, Run cadence, and Notes columns to express this; do not add a new column.
5. Specify Step 5 human inputs:
   • grading rubric, ground truth, golden answer, or rubric + ground truth;
   • author/owner;
   • grounding source dependency;
   • whether source changes require review.
6. Plan Step 6 grader validation without changing the template:
   • record grader type and validation expectation in the registry row's Notes;
   • for LLM-as-judge / Custom rubrics, note that human-labeled hard and borderline cases must validate the judge before baseline scores are trusted;
   • for programmatic checks, note the deterministic check to confirm;
   • for human grading, note reviewer agreement expectations where relevant.
7. Seed Step 7 baseline placeholders in 3 . Run Log only when useful:
   • one placeholder row per eval set;
   • Run type = Baseline;
   • result fields blank;
   • Actionable next step = Validate grader, then run baseline;
   • Status = Open.
8. Apply Step 8 regression partitioning in existing registry fields:
   • capability sets usually Intended use = Both or Regression;
   • most T&S sets are Gate; the slim subset likely affected by model/tool/policy changes can be Both or Regression;
   • set Run cadence using existing dropdown values such as Per-change, Nightly, Weekly, or Milestone-only.
9. Flag Step 10 reusable assets in 4 . Reusable Library:
   • reusable T&S sets;
   • grading rubrics;
   • failure-pattern templates;
   • production-derived edge-case categories when applicable.