eval-harness

Eval Harness

Build a reproducible evaluation harness for LLM pipelines, AI features, or agent workflows. The harness consists of:

• Eval definitions — test cases with inputs, expected outputs, and scoring rubrics
• Runner — executes the pipeline against all test cases
• Scorer — applies rubrics and records results
• Tracker — maintains pass/fail history across runs (via SQL session DB)

When to Use

• Building a new LLM-powered feature and want regressions caught automatically
• Changing prompts and want to confirm no quality degradation
• Demonstrating quality evidence for a shipped AI pipeline
• Setting a quality gate for a CI/CD pipeline

When NOT to Use

Instead of eval-harness	Use
Spot-check one interaction	answer directly
Standard software unit tests (no LLM output)	tdd-workflow skill
Formal red-team safety evaluation	security team involvement required

Eval Directory Layout

.evals/
  <harness-name>/
    config.json          # harness metadata
    cases/               # individual test cases
      01_basic.json
      02_edge_case.json
    rubrics/             # scoring rubrics
      accuracy.md
      format.md
    results/             # run results (auto-generated)
      2024-01-15_run001.json

Workflow

1. Define the eval scope

What pipeline or feature are you evaluating?
What does "good" output look like?
What are the critical failure modes?

2. Write test cases

Minimum viable test suite structure:

Test type	Minimum count
Happy path (well-formed inputs)	5
Edge cases (unusual but valid)	3
Near-miss (close to but not in scope)	3
Adversarial / jailbreak attempts	2

Each test case file:

{
  "id": "tc_01",
  "name": "Basic summarization accuracy",
  "kind": "execution",
  "input": "Summarize this article: [article text]",
  "expected_output": {
    "contains": ["main topic", "key insight"],
    "excludes": ["hallucinated fact"],
    "format": "3-5 sentences"
  },
  "rubric": "accuracy + format",
  "tags": ["happy-path", "summarization"]
}