analyzing-experiment-session-replays

Analyzing experiment session replays

This skill guides you through analyzing session recordings for experiment variants to understand behavioral differences between control and test groups.

When to use this skill

Use this skill when:

• The user asks to analyze session replays for an experiment
• The user wants to understand how users behave differently across experiment variants
• The user asks to compare user behavior between control and test variants
• The user wants qualitative insights to complement experiment metrics
• The user asks questions like "How are users behaving in my experiment?" or "Show me session replays for variant X"

Prerequisites

Before analyzing session replays:

1. The experiment must be launched (not in draft state)
2. Session replay must be enabled for the project
3. Users must have been exposed to the experiment variants
4. The experiment must have a start date

Workflow

1. Get experiment details and feature flag variants

First, retrieve the experiment information and the feature flag variants (source of truth).

Step 1a: Get experiment metadata

You can either:

• Option A: Use the experiment-get tool if you already have the experiment ID from context
• Option B: Query the experiments table via HogQL:

SELECT
    e.id,
    e.name,
    f.key AS feature_flag_key,
    e.start_date,
    e.end_date
FROM system.experiments e
JOIN system.feature_flags f ON f.id = e.feature_flag_id
WHERE e.id = <experiment_id>

From the experiment data, extract:

• feature_flag_key: The feature flag controlling the experiment
• start_date and end_date: The experiment's time range

Step 1b: Get variants from the feature flag

IMPORTANT: Always get variants from the feature flag, NOT from experiment.parameters.feature_flag_variants. The parameters can be out of sync or deprecated. The feature flag is the source of truth.

Query the feature flag to get the current variants:

SELECT filters.multivariate.variants AS variants
FROM system.feature_flags
WHERE key = '<feature_flag_key>'