evaluating

Evaluating

An eval settles one question: does this change what the agent does? Every section below protects that question from the ways it quietly becomes a different, easier question that the same numbers appear to answer.

The engine runs the trials — arcforge eval list | lint | preflight | ab | run | compare | report | audit. What it cannot do is decide whether the thing it measured is the thing you claimed. That decision is this skill.

Phase 1 — Name the claim, then name what varies

Write the claim as one sentence about a decision the agent makes: "with this instruction the agent reproduces the failure before proposing a fix." Vague claims ("the skill helps") cannot be wrong, so they cannot be measured. Then read off what has to differ between the two runs:

The question you are asking	What varies between the runs
Does this instruction change behavior?	the instruction — present vs absent
Is the agent reliable at this task?	nothing; the same run repeated
Does the installed toolkit change outcomes?	the environment — plugins, hooks, project rules

Vary the wrong thing and the number answers the wrong question. A claim about an installed workflow, tested by pasting the instruction text into the prompt, measures the prompt — the installation was never absent in either run.

A claim that bottoms out in an artifact — a file appeared, the exit code was 0, the JSON parsed — is not behavioral. Nothing in it depends on the agent's judgment, so a unit test settles it faster and without sampling noise.

• Done when the claim names one agent decision in one sentence, and you can say which row above you are in and what is different between the two runs.

Phase 2 — Build a trap the baseline falls into

A scenario earns its keep by discriminating. Before running anything, say out loud why the baseline — same prompt, same files, no instruction — will get this wrong. If you cannot finish that sentence, you have no eval yet: you have a task both conditions will pass, and a delta of zero waiting to be misread.