suede-ab-testing

Suede A/B Test Setup

Use this Suede experimentation playbook to design tests that produce statistically valid, actionable results.

The Iron Law

Predeclare three things before a test launches — sample per variant,
minimum duration, and the decision rule — and read the result only once
all three are satisfied. A result read before then is preliminary.
Never a winner.

• Sample per variant: the Sample Size table below, or a calculator run on your actual baseline.
• Minimum duration: 1 full week (day-of-week variation), 2 business cycles (B2B), through paydays (e-commerce) — see the "Minimum Duration Rules" section of references/sample-size-guide.md.
• Decision rule: which metric, at which threshold, decides the call — written down before launch, not after.

Two carve-outs, and only these two:

• A predeclared sequential or always-valid design may look early under its own stopping rule (see "Sequential Testing" in the sample-size guide). Declaring it sequential after the peek does not count.
• A guardrail-triggered stop for harm is a stop, not a winner call. Kill the variant, report no result.

Initial Assessment

Check for .agents/product-marketing.md (or .claude/product-marketing.md, or the legacy product-marketing-context.md) and read it if present — baseline conversion rate, traffic volume, and available tooling decide whether a test is even powerable, and they are usually already written down there.

Then work the intake list under Task-Specific Questions below; ask only what the context file did not already answer.

---

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Example

Weak: "Changing the button color might increase clicks."

Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."