evaluation

Evaluation Frameworks for Agent Systems

Build evaluation systems that catch regressions before deployment and measure agent performance objectively. Agent evaluation differs from traditional software testing because outputs are non-deterministic — the same input can produce different valid outputs. The solution is layered evaluation: deterministic checks for structure and constraints, statistical checks for behavior, and LLM-based checks for subjective quality.

When to Activate

Activate this skill when:

• Designing evaluation pipelines for agent systems
• Building regression tests for agent behavior
• Defining pass/fail criteria for agent outputs
• Choosing between evaluation approaches (deterministic vs. statistical vs. LLM-based)
• Measuring agent performance across versions or configurations

Do not activate this skill for adjacent work owned by other skills:

• LLM-as-judge scoring, pairwise comparison, and rubric generation: advanced-evaluation.
• Designing agent operating loops with evaluation gates: harness-engineering.
• Choosing project pipeline architecture: project-development.
• Browser-based regression QA with fix loops: GStack /qa, /qa-only — those execute tests in Chromium; this skill designs eval suites and metrics.

Core Concepts

Stack evaluation in three layers, each catching different failure classes:

1. Deterministic checks (fastest, most reliable) — Format validation, schema compliance, constraint satisfaction, required field presence, output length bounds. These are binary pass/fail with zero ambiguity.

2. Statistical checks (medium speed, medium reliability) — Aggregate metrics over multiple runs: success rate, average quality score, latency distribution, token usage patterns. Require sample sizes large enough for statistical significance.

3. LLM-based checks (slowest, most nuanced) — Semantic quality assessment, coherence evaluation, instruction following, factual accuracy. See advanced-evaluation for implementation details.