ab

/tokenwise:ab — A/B test a task across model tiers

Run the same task at multiple tiers and compare outputs.

Parse $ARGUMENTS

Expected form: <task description> [--tiers haiku,sonnet,opus,fable]

• The task description is everything before the first -- flag (or the whole string if no flags)
• --tiers haiku,sonnet is the default (skip Opus and Fable by default — Opus is the baseline, Fable is the priciest lane)
• --tiers haiku,sonnet,opus runs the three cheaper tiers
• --tiers ...,fable adds Fable — useful for calibrating the Planning lane (e.g. checking whether a task class really needs it, or whether an override pinning it to sonnet/opus is good enough). Since Fable costs 2× Opus, the cost-confirm step below should flag it

If $ARGUMENTS is empty, ask the user:

What task should I A/B test? Provide a task description (e.g., "rename getCwd to getCurrentWorkingDirectory across the codebase").

Steps

1. Confirm cost upfront:

   A/B test will run this task <N> times (once per tier).
   Estimated cost: $<rough estimate based on task size>.
   Proceed? [Y/n]
   

   Estimate by treating the task as ~10k input + ~1k output per tier and summing. If fable is in --tiers, add one line before the prompt: Note: fable is ~2x Opus's rate and will dominate this estimate.

2. For each tier:

   • Spawn a Task at that tier: Task(description: <task>, subagent_type: "general-purpose", model: <tier>, prompt: <task>)
   • Capture: stdout, input_tokens, output_tokens, duration_ms, errors
   • If the Task tool's model: param is silently overridden (Anthropic Issue #47488), warn user and abort with:

     Cannot A/B test on this Claude Code build — subagent model routing is not honored. See /tokenwise:install probe results.

3. Compute diffs:

   • Structural: line count, character count, token overlap (Jaccard on word sets)
   • File-list diff: if outputs mention modified files, compare the lists
   • Semantic: spawn one more Task at Opus tier with the prompt:

     Compare these N outputs for the task "". Score each from 1-10 on (a) completeness, (b) correctness, (c) clarity. Return a JSON object: {"<tier>": {"completeness": N, "correctness": N, "clarity": N, "overall": N, "notes": "..."}}. Be honest — if two outputs are equivalent, give them the same score.