ab-harness

Claude Code A/B Counterfactual Harness

Problem

The user wants to quantify whether their accumulated ~/.claude/ setup (axioms, lessons, skills, plugins, hooks, auto-memory) actually helps on representative tasks, not just whether artifacts get referenced (that's ecosystem-audit's job).

Naive attempts fail because:

• The default Claude Code run pulls the full ~/.claude context — there's no obvious switch to "turn the setup off."
• Simply cd-ing to a different project doesn't help — global axioms/skills still load.
• Even with a clean env, the project repo's docs/runbooks/, docs/findings/, and in-repo MEMORY.md stay visible — the harness must either accept that leakage (measure marginal ~/.claude value) or scrub repo files too.

Context / Trigger Conditions

• User has completed the ecosystem-audit scan (utilization %) and wants the counterfactual number next ("setup saved X turns / prevented Y pitfalls").
• User says "I want to show colleagues my setup works," "quantify the setup," "A/B test my Claude setup," "replay a task without my memory."
• A project plan references a counterfactual harness step — the canonical measurement stage after (1) usage audit and (2) before (3) cleanup.
• User asks whether a specific skill / lesson / axiom pulls its weight on real work.

Solution

Step 1 — Verify the clean-env mechanism before spending a cent

The knob is CLAUDE_CONFIG_DIR. Point it at an empty dir:

mkdir -p /tmp/claude-empty
CLAUDE_CONFIG_DIR=/tmp/claude-empty claude -p --output-format json \
  "Reply JSON: {has_axioms: <bool>, num_skills_visible: <int>, session_start_hook_fired: <bool>}" \
  < /dev/null

Expected diff vs a normal run (same probe, no env var):

Signal	Normal	Clean (CLAUDE_CONFIG_DIR=/tmp/claude-empty)
skills visible	hundreds	~10 (defaults only)
axioms auto-loaded	true	false
SessionStart hook fired	true	false
project auto-memory (~/.claude/projects/*/memory/MEMORY.md)	injected	empty