experiment-design

Codex host binding

• Treat references to Claude slash workflows as the equivalently named Codex skill.
• Before delegating to any specialist, read the great-pm-runtime skill and the selected packaged role file.
• Treat "invoke", "assign", "delegate", "spawn", and source Agent-tool instructions as a required Codex spawn_agent call with that role and a bounded assignment.
• For the spawn tool's task_name or agent_name identifier, replace canonical role-name hyphens with underscores (query-refiner-pm → query_refiner_pm). Keep canonical names unchanged in prompts, filenames, verdict logs, and user-facing output; this is an internal identifier mapping, not a role rename.
• Store every returned agent identifier. Never call a wait tool until a spawn has returned an identifier, and wait only on identifiers returned by successful spawns.
• If spawn_agent is unavailable or a spawn fails, report BLOCKED; do not impersonate the specialist or wait on an empty agent set.
• Resolve bundled paths from the installed GreatPM plugin root.
• Ignore Claude-only model aliases, colors, turn limits, and tool allowlists.
• Preserve GreatPM human gates, governance, state, and reporting contracts.

Experiment Design — playbook

Experiments produce useful answers only when they are honestly designed. Underpowered tests produce noise; peeking produces false positives; guardrail-blind tests produce shipped regressions.

1. Hypothesis — one sentence with a mechanism

If we do <change>, <primary metric> will move by <effect size> because <mechanism>.

• The metric is real and pre-defined (it is in the metrics plan).
• The effect size is the MDE — the minimum effect that would matter for the business. Smaller MDE = bigger sample required.
• The mechanism is why this should work. If you cannot articulate the mechanism, the result will not generalise.

2. Test types — pick by what you can randomize

• A/B test — randomize users into control vs treatment. Default for feature changes.
• Multivariate — multiple variants at once (A/B/C/D). Beware: inflates multiple-comparison risk; correct alpha or use one decision metric.
• Holdout — keep a slice of users in pure control long-term, even after launch. Useful for measuring cumulative effect.
• Switchback — for marketplace / two-sided contexts where user-level randomization leaks. Alternate the treatment in time windows by region or segment.