instruction-eval

Condition A/B

Change one condition an agent runs under, run the same prompts before and after, and show the difference.

The conditions surrounding an agent have no verification. You can read code and tests will catch a regression, but a few lines added to instructions or a reference doc dropped in a directory only ever get judged on whether they sound reasonable. Even the person who put them there has no idea whether they change behavior. This skill replaces that guess with an observation.

This runs on Claude Code. Both arms execute as claude -p subprocesses, so the CLI has to be available.

Who writes what

The report holds content from two sources, visually separated in the HTML. Never hand-write what the script produces, since transcribing only introduces errors.

Automatic (scripts)	LLM (insights.json)
Experiment setup: n, model, both directory paths	What you changed
Prompt text	The claim about what changes
Answer text (every arm, every rep)	Differences observed per prompt
Metric medians and deltas	Metric interpretation
Run anomalies (errors, permission denials)	Conclusion

Procedure

1. Design and build the conditions

Pick the axis first. What you compare against what determines everything else. Only what the user mentioned is a candidate, and even when it reads as obvious you confirm it in step 2.

Anything can be the axis: instruction text, reference material instructions point at, MCP tools and permissions, hooks, skill files. The script only sees two directories, so all of these get handled identically.

baseline is the working directory in the before state, variant the after state.

The purpose decides the direction.

	Verifying an addition	Building a case for deletion
baseline	axis absent	axis present (status quo)
variant	axis present	axis absent
What you watch	does it do what the axis asks	does the failure it prevented show up

Deletion is the more common need. Instructions only grow in one direction and nothing ever makes the case for removing them, while an ineffective instruction still rides along on every request, spending context and diluting the signal of the instructions that do work. Judge deletion asymmetrically, though. The bar is not "delete when there's no evidence of effect" but "delete only when removing it is confirmed to produce no failure." A symmetric bar at small sample sizes will delete rules on noise.