experiment

Skill: /experiment — OpenXP Experimentation Platform

Purpose

Multi-mode skill for the full experiment lifecycle — from design through analysis to ship/no-ship decision. Orchestrates experiment agents and calls coded statistical helpers from helpers/stats/experiment_stats/ instead of improvising Python.

When to Use

Invoke as /experiment [mode] or trigger on experiment-related intents:

• "I want to run an experiment"
• "Analyze this A/B test"
• "Did this experiment work?"
• "What's the power for this test?"

Modes

/experiment design

Purpose: Create a pre-registered experiment config. Agent: agents/experiments/experiment-designer.md Flow:

1. Run Experiment Brief skill to capture hypothesis, north star, guardrails
2. Invoke Experiment Designer agent
3. Output: experiments/{slug}/experiment.yaml (from templates/experiment.yaml) Checkpoint: Config review (Type B — skippable with --just-do-it)

/experiment power

Purpose: Power analysis + duration estimation. Flow:

1. Read experiments/{slug}/experiment.yaml for metric type, baseline, MDE
2. Call helpers/stats/experiment_stats/power.py:
   • Proportion metric → power_proportion(baseline_rate, mde)
   • Continuous metric → power_mean(baseline_mean, baseline_std, mde)
3. Call duration_estimate(total_sample, daily_traffic, allocation)
4. Update experiment.yaml with computed values (sample_size, duration, viable)
5. If NOT_VIABLE → suggest /causal select as alternative Checkpoint: Power viability (Type C — NOT_VIABLE fires mandatory checkpoint)

/experiment analyze

Purpose: Run statistical tests on experiment data. Agent: agents/experiments/experiment-analyzer.md Flow:

1. Read experiments/{slug}/experiment.yaml for pre-registered config
2. SRM Gate (mandatory first step):

   from helpers.stats.experiment_stats import srm_check
   # Positional lists ONLY — do not pass dicts.
   # First arg: observed counts per variant (order must match expected_ratios).
   # Second arg: expected allocation ratios, summing to 1.0.
   result = srm_check([4218, 4196], [0.5, 0.5])
   # result = {"chi2_stat": 0.058, "p_value": 0.81, "verdict": "PASS", ...}
   if result["verdict"] == "BLOCK":
       # HALT — do not proceed to treatment effect analysis
   

3. Treatment effect analysis using coded helpers:

   from helpers.stats.experiment_stats import welch_test, proportion_test, ratio_metric_test
   # Select based on metric type from experiment.yaml
   if metric_type == "proportion":
       result = proportion_test(c_success, c_n, t_success, t_n)
   elif metric_type == "continuous":
       result = welch_test(control_values, treatment_values)
   elif metric_type == "ratio":
       result = ratio_metric_test(num_c, den_c, num_t, den_t)
   

4. Effect size: cohens_d(control, treatment)
5. Multiple comparisons: adjust_pvalues(all_p_values, method="holm")
6. Guardrail checks against thresholds from experiment.yaml
7. Segment analysis (Simpson's paradox check)
8. Output: experiments/{slug}/working/analysis_results.json Checkpoint: SRM gate (Type C — BLOCK halts everything)