experiment-design-framework

Advertising Experiment Design Framework

Purpose

Provide a rigorous, practical framework for designing and interpreting advertising experiments. Move from "I think this works" to "I know this works, and here's the data." Most ad optimization is observational — experiments let you prove causation.

When to Use This Skill

Invoke when user mentions:

• A/B testing: "How do I A/B test my ads?"
• Statistical significance: "Is this result significant?"
• Sample size: "How many conversions do I need?"
• Test duration: "How long should I run this test?"
• Incrementality: "Is this channel actually driving sales?"
• Holdout test: "What would happen if I turned off this campaign?"
• Geo lift: "How do I test a campaign's true impact?"
• Multi-variate: "Can I test multiple things at once?"
• Learning phase: "How do I test without wasting budget?"

---

Part 1: Experiment Types

Overview Matrix

Experiment Type	Complexity	Cost	Statistical Rigor	Best For
A/B Test (Split Test)	Low	Low	Medium-High	Creative, copy, landing pages
Multi-Variate Test (MVT)	Medium	Medium	Medium	Multiple creative elements simultaneously
Holdout Test	Low	Low-Medium	High	Measuring incrementality of a campaign
Geo Lift Test	High	High	Highest	Measuring true channel contribution
Pre/Post Test	Low	Low	Low	Rough directional signal only
Conversion Lift (Meta/Google)	Medium	Medium	High	Platform-provided incrementality

When to Use Each Type

QUESTION: What are you trying to learn?

├── "Which creative/copy/CTA works better?"
│   └── A/B Test (or MVT if testing multiple elements)
│
├── "Is this campaign actually driving incremental sales?"
│   └── Holdout Test (simplest) or Conversion Lift Study
│
├── "What's the true ROI of this channel?"
│   └── Geo Lift Test (gold standard) or Holdout Test
│
├── "Should I change my bid strategy?"
│   └── A/B Test with Campaign Budget Optimization
│       (run both strategies simultaneously, same audience split)
│
├── "Which audience performs better?"
│   └── A/B Test with audience splitting
│       (Meta: split test feature; Google: experiments)
│
└── "What's the best combination of headline + image + CTA?"
    └── Multi-Variate Test (need high traffic volume)