calibrate

Validate agents and skills by measuring outputs against synthetic problems with defined ground truth. Primary signal: calibration bias — gap between self-reported confidence and actual recall. Well-calibrated agent reports 0.9 when it finds ~90% of issues. Miscalibrated: reports 0.9, finds 60%.

Calibration data drives improvement loop: systematic gaps → instruction updates; persistent overconfidence → adjusted re-run thresholds in MEMORY.md.

NOT for: static routing overlap analysis (use /foundry:audit); manually reviewing skill output quality (use /develop:review (requires develop plugin)).

• $ARGUMENTS: parse --flags first, then resolve remaining tokens as scope targets

  Flags (order independent):

  • --fast — 3 problems per target (default when neither pace flag passed)
  • --full — 10 problems per target; mutually exclusive with --fast
  • --ab-test — also run general-purpose baseline and report delta metrics; requires benchmark (default --fast if no pace flag); mutually exclusive with --apply
  • --apply — apply proposals: with --fast/--full: run benchmark then immediately apply; without pace flag: skip benchmark, apply proposals from most recent past run; mutually exclusive with --ab-test
  • --skip-gate — suppress follow-up gate; for programmatic callers
  • --local — resolve target agent/skill files from source tree (plugins/*/) instead of installed plugin cache; for plugin-dev workflows where local edits aren't yet installed; sets LOCAL_MODE=true in all pipeline spawns

  Mutual exclusion validation (check before any work):

  • --ab-test + --apply together → hard error: "--ab-test and --apply are mutually exclusive. Pass one or neither."
  • --fast + --full together → hard error: "Pass --fast or --full, not both."
  • --ab-test without pace flag → default --fast silently (no error)

  Unsupported flag check — after all supported flags extracted (--fast, --full, --ab-test, --apply, --skip-gate, --local, --keep), scan $ARGUMENTS for remaining --<token> tokens. If found: print ! Unknown flag(s): `--<token>`. Supported: `--fast`, `--full`, `--ab-test`, `--apply`, `--skip-gate`, `--local`, `--keep`. then invoke AskUserQuestion — (a) Abort (stop, re-invoke with correct flags) · (b) Continue ignoring (skip unknown flags, proceed). On Abort: stop.

  Legacy positional tokens (ab, apply, fast, full) — hard error: print migration hint and stop. Example: "ab removed — use --ab-test flag: /calibrate curator --ab-test."

  Scope tokens (positional, space-separated — defaults to all):

  • all — all agents + relevant skills + routing + communication + all rules
  • agents — all agents only (full agent list in modes/agents.md)
  • skills — calibratable skills only (/audit and others per modes/skills.md; /oss:review (requires oss plugin) excluded — requires live GitHub PR)
  • routing — routing accuracy test: measures how accurately general-purpose orchestrator selects correct subagent_type for synthetic task prompts (not per-agent quality benchmark; included in all)
  • communication — handover + team protocol compliance: runs foundry:curator against synthetic agent responses and team transcripts with injected protocol violations (missing JSON envelope, missing summary, AgentSpeak v2 breaches); included in all
  • rules — rule adherence test: for each global rule file (no paths:) and each path-scoped rule when matching file is in context, generates synthetic tasks that should trigger rule's key directives, measures whether general-purpose agent with rule loaded correctly applies them; reports rules that are ignored, misapplied, or redundant; included in all
  • plugins — all agents + calibratable skills from all plugins/*/ directories (union of all plugin-namespaced agents and calibratable skills)
  • <plugin-name> — tier 2: bare plugin directory name (e.g. oss, foundry, research, develop) auto-resolved when token matches plugins/<name>/ directory; calibrates all agents + calibratable skills in that plugin
  • <agent-name> — tier 3: single agent (e.g., foundry:sw-engineer); also accepts bare name (e.g. sw-engineer) and resolves via plugins/*/agents/<name>.md
  • /foundry:audit — single skill (pass any calibratable skill name; /oss:review (requires oss plugin) accepted but excluded per modes/skills.md)
  • Multiple scope tokens — space-separated; calibrates union of resolved targets: oss research, agents skills, curator shepherd; each token resolved through same tier hierarchy as /audit scope tokens (reserved keywords first, then plugin-dir lookup, then agent/skill file search)