What is eval-harness?

A testing framework for Claude Code sessions that checks whether an AI agent completes tasks as expected. It applies eval-driven development, meaning you define expected behavior first and test it throughout development.

Use it to define capability tests, regression test suites, and reliability measurements, including pass@k—the rate at which an agent succeeds across repeated attempts. It can also compare performance between model versions.

It replaces guesswork about agent reliability with repeatable pass/fail checks. It also helps detect when a prompt or model change breaks behavior that previously worked.

eval-harness — Skill for Claude Code by KratosAR

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add KratosAR/claude-code-arsenal --skill eval-harness

Clone the repo

git clone --depth 1 https://github.com/KratosAR/claude-code-arsenal

Made for: Claude Code.

Or install arsenal-core, the plugin that ships this one along with the rest of its 43 skills, 15 agents.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/kratosar/claude-code-arsenal/eval-harness.svg)](https://agentmods.dev/skills/kratosar/claude-code-arsenal/eval-harness)

Your own site

<a href="https://agentmods.dev/skills/kratosar/claude-code-arsenal/eval-harness"><img src="https://agentmods.dev/badge/skills/kratosar/claude-code-arsenal/eval-harness.svg" alt="Measured on agentmods" height="20"></a>

Per session 19 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 1,647 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin 84% copy Near-identical to another mod in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00019	$0.01647
Opus 5	$0.00010	$0.00823
Sonnet 5	$0.00004	$0.00329
Haiku 4.5	$0.00002	$0.00165

Measured 7d ago against content hash 5aa3cf00a03b, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-07, from the pricing page.

Security

Grade A, and why

eval-harness scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 7d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

Origin

This is a copy

84% identical to eval-harness — 54 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.

arsenal-core/skills/eval-harness/SKILL.md · 298 lines

How it starts

The opening of the file, as written. The whole thing — 298 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Eval Harness Skill

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.

When to Activate

Setting up eval-driven development (EDD) for AI-assisted workflows
Defining pass/fail criteria for Claude Code task completion
Measuring agent reliability with pass@k metrics
Creating regression test suites for prompt or agent changes
Benchmarking agent performance across model versions

Philosophy

Eval-Driven Development treats evals as the "unit tests of AI development":

Define expected behavior BEFORE implementation
Run evals continuously during development
Track regressions with each change
Use pass@k metrics for reliability measurement

Eval Types

Capability Evals

Test if Claude can do something it couldn't before:

[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
  - [ ] Criterion 1
  - [ ] Criterion 2
  - [ ] Criterion 3
Expected Output: Description of expected result

Regression Evals

Ensure changes don't break existing functionality:

[REGRESSION EVAL: feature-name]
Baseline: SHA or checkpoint name
Tests:
  - existing-test-1: PASS/FAIL
  - existing-test-2: PASS/FAIL
  - existing-test-3: PASS/FAIL
Result: X/Y passed (previously Y/Y)

Grader Types

1. Code-Based Grader

Deterministic checks using code:

# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"

# Check if tests pass
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"

# Check if build succeeds
npm run build && echo "PASS" || echo "FAIL"

2. Model-Based Grader

Use Claude to evaluate open-ended outputs:

[MODEL GRADER PROMPT]
Evaluate the following code change:
1. Does it solve the stated problem?
2. Is it well-structured?
3. Are edge cases handled?
4. Is error handling appropriate?

Score: 1-5 (1=poor, 5=excellent)
Reasoning: [explanation]

Read the full file on GitHub · 298 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

7d ago First seen · 298 lines · 19 tokens per session scan A 5aa3cf00a03b

Subscribe to this mod's changes

eval-harness is a skill published in the GitHub repository KratosAR/claude-code-arsenal (2 stars, last pushed 21d ago), licensed MIT. It adds 19 tokens to every session and 1,647 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. It is 84% identical to eval-harness, differing in 54 lines, and is treated as a copy.

Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

neuron-evaluation-engineer

Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…

neuron-core/neuron-ai · 77 tokens

eval-harness