output-eval-audit

Auditing an Eval Suite

Overview

Audit your eval suite to determine whether it actually catches real failures. This skill provides a structured diagnostic that identifies gaps in error analysis, evaluator design, judge validation, and dataset coverage, with concrete remediation steps for each finding.

When to Use

• Inheriting an eval suite from another team or developer
• Suspecting that evals pass but production quality is poor
• After switching models, rewriting prompts, or changing pipeline logic
• Periodic health check (quarterly or after major releases)

Step 1: Gather Artifacts

Read the eval infrastructure files for the workflow being audited:

src/workflows/<workflow_name>/
├── tests/
│   ├── datasets/           # YAML dataset files
│   │   ├── *.yml
│   │   └── ...
│   └── evals/
│       ├── evaluators.ts   # Evaluator definitions
│       ├── workflow.ts      # Eval workflow definition
│       └── *.prompt         # Judge prompt files

Inventory what exists:

Artifact	File(s)	Count
Evaluators	tests/evals/evaluators.ts	?
Eval workflow	tests/evals/workflow.ts	? entries in evals array
Judge prompts	tests/evals/*.prompt	?
Datasets	tests/datasets/*.yml	?
Datasets with ground_truth	? of above	?
Datasets with last_output	? of above	?

If any of these are missing entirely, note it and skip to "Starting From Zero" at the bottom.

Step 2: Run the Diagnostic

Evaluate each of the four areas below. For each, assign a status:

• Pass — Meets the standard
• Warn — Partially meets the standard, improvements needed
• Fail — Does not meet the standard, significant risk

---

Area 1: Error Analysis Grounding

Question: Were the evaluators derived from observed failure modes in real workflow traces?

Check:

• Do failure categories exist (documented in a file, comments, or commit history)?
• Does each evaluator map to a specific failure category?
• Or are evaluators measuring generic qualities ("quality score", "overall rating")?