output-dev-eval-testing

Offline Evaluation Testing

Overview

The @outputai/evals package provides an offline evaluation framework for testing workflow quality using datasets and evaluators. This is complementary to the runtime evaluator() from @outputai/core:

Aspect	Runtime Evaluators (@outputai/core)	Offline Eval Tests (@outputai/evals)
When	During workflow execution	After execution, at test time
Where	evaluators.ts in workflow folder	tests/evals/ in workflow folder
Purpose	Live quality scoring with confidence	Dataset-driven pass/fail verification
Triggered by	Workflow orchestration	output workflow test CLI command
Returns	EvaluationBooleanResult, etc.	Verdict helpers (pass/partial/fail)

Use offline eval testing when you want to validate workflow behavior against known datasets, build regression test suites, or assess subjective quality with LLM judges.

When to Use This Skill

• Creating files in tests/evals/ or tests/datasets/
• Writing evaluators that use verify() from @outputai/evals
• Creating YAML dataset files for test cases
• Building eval workflows with evalWorkflow()
• Running output workflow test commands
• Setting up ground truth data for evaluators

Directory Structure

Add a tests/ directory inside the workflow folder:

src/workflows/{workflow_name}/
├── workflow.ts
├── steps.ts
├── evaluators.ts          # Runtime evaluators (optional)
├── types.ts
└── tests/
    ├── datasets/
    │   ├── happy_path.yml
    │   └── edge_case.yml
    └── evals/
        ├── evaluators.ts  # Offline eval test evaluators
        ├── workflow.ts     # Eval workflow definition
        └── [email protected]  # LLM judge prompts (optional)

Creating Evaluators with verify()

Import verify and Verdict from @outputai/evals (not @outputai/core):

// tests/evals/evaluators.ts
import { verify, Verdict } from '@outputai/evals';
import { z } from '@outputai/core';