statistical-analysis-and-experiments

Statistical Analysis and Experiments

Overview

A statistical claim is a decision under uncertainty. The analysis has to say how big the effect is, how sure we are, and what could make it wrong — not just whether p < 0.05. The statistics code is production code too: it gets tests.

Core principle: decide the design (metric, unit, sample size, stopping rule) before looking at results; report effect size with a confidence interval; treat every causal-sounding sentence as a claim that needs a design behind it.

1. Describe before you test

• Report mean and median for business metrics; a large gap means skew and the mean alone misleads.
• Use percentiles (p5, p25, p50, p75, p95, p99) and the IQR for skewed data, standard deviation for roughly symmetric data.
• State shape, bounds (a floor at zero, a ceiling at 100%) and how outliers were handled. Never drop outliers silently: classify them as data errors (fix or exclude, with a count), genuine extremes (keep, use robust statistics) or a different population (segment).

2. Design an experiment before it runs

Write these down, in the task or your closing message, before analysing anything:

Item	Example
Hypothesis	New checkout raises conversion
Primary metric (one)	Orders / visitors, per user
Guardrails	Refund rate, page latency
Randomisation unit = analysis unit	User (not session, not page view)
Minimum detectable effect	+1.0 percentage point on a 10% baseline
α, power	0.05 two-sided, 0.80
Sample size and duration	Computed below; whole weeks to cover weekly cycles
Stopping rule	Analyse once at the planned sample size

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.11, 0.10)
n_per_arm = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8, ratio=1.0)

If the traffic cannot reach n_per_arm in a reasonable time, say so before the test runs — an underpowered test produces noise and exaggerated "wins".