causal-inference

Causal Inference (experiments, DiD, synthetic control, quasi-experiments)

The question is always about a counterfactual

Every causal claim compares the observed world to one that did not happen: what would this group's outcome have been without the treatment? Method choice is only about how that counterfactual is built — randomisation by design, quasi-experiments by assumption. State the assumption; show evidence for it.

randomised (A/B, geo holdout) → counterfactual = the control arm; assumption: valid randomisation
DiD                           → = treated group's own pre-trend continued; assumption: parallel trends
synthetic control             → = weighted mix of untreated units matching the treated pre-period
RDD                           → = units just the other side of a threshold; assumption: continuity
IV                            → = variation from an instrument; assumptions: relevance + exogeneity

Randomisation first (the gold standard)

Randomise at the level the treatment is delivered: users for in-product changes, geos for media, markets for pricing. Fix the unit, primary metric, analysis window and minimum detectable effect before launch, with power computed from the metric's own variance. Geo holdouts are the marketing workhorse because ads cannot be reliably withheld per individual: pair or stratify geos on pre-period level and trend, and hold out enough population to be powered (a token 5% holdout rarely detects a real media effect).

Difference-in-differences

effect = (Y_treated,post − Y_treated,pre) − (Y_control,post − Y_control,pre)
assumption: absent treatment, both groups' outcomes would have moved in parallel

DiD subtracts the control's change from the treated group's change, cancelling level differences and shocks common to both. Parallel trends concerns an unobserved counterfactual: never provable, only falsifiable. Check it — plot both series over many pre-periods (not two points) and run an event-study specification with leads and lags, confirming every pre-treatment lead is ≈0. Breakages: the treated unit was chosen because it was already moving (Ashenfelter's dip), a shock hit one group only, or timing is staggered (needs a staggered-adoption estimator, not naive two-way fixed effects). Cluster errors at the treatment unit; treating each user-day as independent shrinks the interval to fiction.