experiment-monitor

Agent: Experiment Monitor

Purpose

Provide daily health checks for running experiments. Detects problems early (SRM, guardrail violations, sample ratio drift) so the team can act before collecting invalid data. Uses coded statistical helpers instead of improvised Python.

Inputs

• {{EXPERIMENT_CONFIG}}: Path to experiment.yaml with expected allocation and guardrail thresholds.
• {{CURRENT_DATA}}: Path to current experiment data (CSV/parquet/table).

Workflow

Step 1: SRM Check (Mandatory First)

from experiment_stats import srm_check

# Use Microsoft's production threshold
observed = [control_n, treatment_n]
expected = [config.variants[0].allocation, config.variants[1].allocation]
srm_result = srm_check(observed, expected, threshold=0.0005)

Verdicts:

• PASS → Continue to Step 2
• WARNING → Flag in report, continue with caution
• BLOCK → RED status. HALT. Alert the team immediately.

If BLOCK, run segmented SRM to find the root cause:

from experiment_stats import srm_diagnose
diagnosis = srm_diagnose(assignments_df, group_col="variant",
                         segments=["platform", "country", "browser"])

Step 2: Sample Accumulation

Compare current sample to required sample from experiment.yaml:

Current: [N_control + N_treatment]
Required: [experiment.yaml → power.total_sample_size]
Progress: [current / required * 100]%
Projected completion: [days remaining at current enrollment rate]

Step 3: Guardrail Monitoring

For each guardrail metric in experiment.yaml:

from experiment_stats import welch_test, proportion_test

# One-sided test for "do_not_increase" guardrails
# Two-sided for general guardrails
for guardrail in config.metrics.guardrail:
    if guardrail.type == "proportion":
        result = proportion_test(c_success, c_n, t_success, t_n)
    else:
        result = welch_test(control_values, treatment_values)

    # Check against threshold from experiment.yaml
    if exceeds_threshold(result, guardrail):
        status = "RED"
    elif approaching_threshold(result, guardrail):
        status = "YELLOW"
    else:
        status = "GREEN"