design-experiment

design-experiment

Purpose

Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness are modes.

When to use / not applicable

Use after a hypothesis has an operational construct and discriminating prediction. Not applicable when the hypothesis or measurable outcome is absent.

Input contract

mode_contracts:
  factorial: &experiment_input
    required: [falsifiable_hypothesis, outcome, factors, constraints]
    optional: [baseline, candidate_models, budget, robustness_axes]
    constraints: [analysis_plan_must_be_preregistered]
  ablation: *experiment_input
  comparison: *experiment_input
  scaling: *experiment_input
  robustness: *experiment_input

Execution protocol

Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.

1. You MUST load skill identify-variables to operationalize the outcome, factors, controls, and their functional roles. You MUST load skill enumerate-dimension-values to define admissible levels or perturbation values. If the resource envelope or feasibility constraints cannot support an executable design, consider analyze-constraints-readiness before committing to a mode.
2. Choose factorial, ablation, comparison, scaling, or robustness mode.
3. You MUST load skill specify-metrics to preregister metrics, estimands, directionality, and decision thresholds. You MUST load skill estimate-sample-size to derive the sample or repetition requirement. You MUST load skill select-statistical-method to select the inference or estimation method before observing outcomes; specify power and the stopping rule. If the proposed metric or validator may share artifacts, labels, or assumptions with the system under test, consider audit-validator-independence before freezing the analysis plan.
4. You MUST load skill construct-design-matrix to construct the runnable matrix for the selected mode. You MUST load skill design-randomness-protocol to define seeds, repetitions, and propagation rules. You MUST load skill specify-execution-environment to capture interpretation-relevant hardware, software, data, configuration, and versions. You MUST load skill specify-reproducibility-protocol to define and test the intended reproduction level. You MUST load skill optimize-design-under-budget to select a feasible information-efficient design under the declared resource envelope; include resource and failure checks. Once the design has produced observations and the task changes from planning to inference, analyze-experiment-results may be the better next tactic.