output-dev-evaluator-function

Creating Evaluator Functions

Overview

This skill documents how to create evaluator functions in evaluators.ts for Output SDK workflows. Evaluators are used to assess quality, validate outputs, and provide confidence-scored judgments about workflow results.

When to Use This Skill

• Implementing quality assessment for workflow outputs
• Adding validation logic with confidence scores
• Creating LLM-powered content evaluation
• Building reusable evaluation components

File Organization

Option 1: Flat File (Default)

For smaller workflows, use a single evaluators.ts file:

src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts
├── evaluators.ts    # All evaluators in one file
├── types.ts
└── ...

Option 2: Folder-Based (Large workflows)

For larger workflows with many evaluators, use an evaluators/ folder:

src/workflows/{workflow-name}/
├── workflow.ts
├── steps.ts
├── evaluators/      # Evaluators split into individual files
│   ├── quality.ts
│   ├── accuracy.ts
│   └── completeness.ts
├── types.ts
└── ...

Component Location Rules

Important: evaluator() calls MUST be in files containing 'evaluators' in the path:

• src/workflows/my_workflow/evaluators.ts ✓
• src/workflows/my_workflow/evaluators/quality.ts ✓
• src/shared/evaluators/common_evaluators.ts ✓
• src/workflows/my_workflow/helpers.ts ✗ (cannot contain evaluator() calls)

Activity Isolation Constraints

Evaluators are Temporal activities with strict import rules to ensure deterministic replay.

Evaluators CAN import from:

• Local workflow files: ./utils.js, ./types.js, ./helpers.js
• Local subdirectories: ./lib/helpers.js
• Shared utilities: ../../shared/utils/*.js
• Shared clients: ../../shared/clients/*.js
• Shared services: ../../shared/services/*.js

Evaluators CANNOT import:

• Other evaluator files (activity isolation)
• Step files
• Workflow files

Example of WRONG imports:

// WRONG - evaluators cannot import other evaluators
import { otherEvaluator } from '../../shared/evaluators/other.js'; // ✗
import { anotherEvaluator } from './other_evaluators.js'; // ✗