experiment-analyzer

Experiment Analyzer

Analyze A/B tests and experiments — measure impact, check statistical validity, break down results by segment, and tell you which variant won (and by how much).

When to Use

• "Did the new checkout design improve conversion?"
• "Analyze the 'blue button vs green button' experiment"
• "Is the pricing page experiment statistically significant?"
• "Break down the experiment results by device type"
• "Which variant performed better for enterprise users?"
• "Should we ship the new onboarding flow?"

Mental Model

An experiment compares two or more variants against a control. The goal is to determine:

1. Direction: Which variant performed better?
2. Magnitude: By how much? (+5%? +20%?)
3. Confidence: Is the difference real or noise? (statistical significance)
4. Segments: Does it work for everyone, or only certain users?
5. Side effects: Did the variant improve conversion but increase errors?

Use free fullstory:discover_groups and fullstory:get_opportunity_stats for live signals. Discovery accepts one scope at a time (page_id, domain, app_name, segment_id, or funnel_id); use a segment for device/user filters and the live schema for time ranges. Use computed metrics for exact custom windows.

Workflow

Step 1: Define the experiment

Ask the user:

• What's being tested? (e.g., checkout flow redesign)
• What's the success metric? (e.g., checkout completion rate)
• When did the experiment start? (so you can scope the time range)
• How are users split? (50/50 random? By user property? By feature flag?)

If users are split by a user property (e.g., experiment_group = "control" or "variant"), use segments. If split by a feature flag that maps to a custom event, filter by that event.

Step 2: Build metrics per variant

For user-property splits (experiment_group property):

fullstory:build_segment("users in experiment group 'control'") → seg_control
fullstory:build_segment("users in experiment group 'variant'") → seg_variant
fullstory:build_metric(query="checkout completion rate", output_type="single_number")
fullstory:update_metric(metric_id, segment_id=seg_control)
fullstory:compute_metric(metric_id) → control_result
fullstory:update_metric(metric_id, segment_id=seg_variant)
fullstory:compute_metric(metric_id) → variant_result