local-llm-mcp-server: Skill for Claude Code

Causal Experiment Controls

A raw accuracy/loss number almost never answers "does the mechanism do X". Design the metric so an artifact CANNOT produce a positive. Bake these controls into the eval report from the first run — not after a surprising result.

Canonical diagnostic order (run every time, in this order)

1. Artifact hypothesis first (see artifact-first-analysis) — do not interpret yet.
2. Constant-baseline check — majority-class / modal-token accuracy + intervention-OFF acc.
3. Correct-vs-wrong state check — Δ = acc(correct) − acc(scrambled), and how many predictions change when the state is swapped.
4. Raw prediction histogram — unique-count + top-k tokens (mode-collapse detector).
5. Label distribution comparison — prediction distribution vs true label distribution.
6. Only then mechanistic interpretation — minimal, scoped, laddered. The report block below emits steps 2–5 in one line so you can never skip them.

The five mandatory controls

1. Content-sensitivity ablation (the core one). Run the model with the REAL input to the mechanism and with a SCRAMBLED input (wrong instance's value, shuffled, or noise), same everything else. The real metric is Δ = acc(real) − acc(scrambled). Δ≈0 ⇒ the mechanism's content is causally inert, no matter how high acc(real) looks. Distinguish two effects and report BOTH:

   • presence-effect: does turning the intervention ON vs OFF change outputs?
   • content-effect: does changing WHAT the intervention carries change outputs? Presence without content = a constant bias, not information use.

2. Label-frequency / majority baseline. Compute max_class_freq and the accuracy of "always predict the modal token/label". If your metric ≈ this, you have learned nothing. Report the baseline NEXT TO the metric, always.

3. Mode-collapse detection. Report the number of UNIQUE predictions across the eval set and the top-k prediction histogram. uniq_pred==1 (or dominated by one token) ⇒ collapse; the accuracy is just that token's label frequency.