instruction-calibration

For reusable skill structure, also load an applicable skill-authoring skill.

Define the comparison

• Start with a real request, observed failure, or durable correction. Name the consequential choice to change and what must stay unchanged.
• Use the natural request, its closest decision boundary, and an unrelated non-use case. Do not put the desired solution into the task.
• Separate remembered knowledge from productive performance. A closed-book probe can expose stale assumptions, but the working baseline should have the docs and source normally available. Verify generated work against the active API and execute relevant checks. Plausible output is not correctness.

Control the layers

• Inspect current harness help. Start with few local instruction layers, record what remains, then restore docs access, repository context, and skills deliberately. These discovery runs are not isolated instruction comparisons when several variables change.
• For Pi, pi -nc -ns -ne disables discovery, not an agent's ability to read instruction files. Inspect actual reads and loaded prompt layers. Exclude contaminated baselines or restrict instruction-file access equally in both arms.
• Compare fresh sessions with identical model, reasoning, tools, cwd, task, and available files. Change only the candidate instruction. Keep follow-ups and reloads for exploration, not clean baseline evidence.
• Test the one-shot output before coaching. Do not credit corrections from a follow-up to the original instruction.
• Finish with the normal worker environment to check skill selection and conflicting layers. Treat that as integration evidence, not an isolated comparison.

Run with the agreed supervision

Use visible sibling sessions when available, preserving cwd and user focus. Consult current harness controls rather than embedding launch recipes. Keep only active comparisons open.

For an interactive comparison, let each meaningful run finish and pause for the user to inspect it before changing the candidate. When the user delegates evaluation, run, judge, refine, and clean up autonomously. Report evidence at completion instead of requiring approval between probes. Human UX preference still needs the user's judgment.