experiment-result-reader

Experiment result reader

A/B tests die from two failure modes that have nothing to do with the variant: reading the result before the sample is big enough, and reading the result without controlling for the wrong thing. This skill encodes the discipline analysts use to avoid both.

When NOT to run this

• The user wants to design a new experiment (sample-size estimation, MDE, variant logic). That's a different skill: design vs read.
• The experiment isn't declared in event-schema.yaml and exposure events aren't being fired. There's nothing to read; nudge the user toward instrumenting the experiment first via event-schema-author or their platform of choice.
• The conversion metric is more than three steps removed from exposure (e.g. "exposure → 30-day retention → upgrade → renewal"). Sequential / cohort-windowed analysis is genuinely harder; this skill stays at the per-period rate level and is honest about it.

Method

Phase 1. Verify the setup

Before reading anything, confirm two facts:

1. The experiment is declared. Read event-schema.yaml's experiments: section. The experiment should have a name, variants list, and ideally an intent. If it isn't declared, ask the user where the variant assignments live and whether the exposure event is firing reliably.
2. Exposure events are landing for every variant. Query the analytics platform (with Clamp: events.list(name="$exposure", group_by="variant") or whichever event name is canonical for the platform; Mixpanel uses $experiment_started, Amplitude uses $exposure). You should see roughly even counts per variant if assignment is unbiased; a heavy skew is a setup bug, not a result.

If either check fails, stop and surface the issue. Don't compute lift on a broken setup.

Phase 2. Pull the four numbers

For each variant, get:

• Exposed users: unique visitors who fired the exposure event for that variant
• Converted users: unique visitors from the exposed set who fired the conversion event in the same period