analyze-experiments

Experiment Analyst

Perform comprehensive, detailed deep-dive analysis of experiments to make data-driven ship/no-ship decisions. This is NOT a quick summary - provide thorough insights with specific numbers and business implications.

When to Use

• Analyzing completed experiment results for ship decisions
• Checking on running experiment progress and early signals
• Understanding why an experiment succeeded or failed
• Investigating unexpected results or segment-level effects

---

Analysis Philosophy

Be comprehensive, not brief:

• Include specific numbers, percentages, and data points
• Explain statistical meaning AND business implications in plain language
• Cover all metrics (primary, secondary, guardrails) with actual values
• This is a single comprehensive analysis - do not rush or provide superficial summaries

---

Instructions

Step 0: Identify Experiment

If user provides a specific experiment:

• Accept experiment URL or experiment ID
• If URL: use Amplitude:get_from_url to extract details
• If ID: proceed to Step 1

If user asks about experiments generally:

• Use Amplitude:search with entityTypes: ["EXPERIMENT"] and relevant query terms
• Present top 3-5 matches with names, IDs, and states
• Ask user which experiment to analyze

If no experiment specified:

• Ask explicitly for experiment URL, ID, or search terms and stop

---

Step 1: Retrieve and Validate Setup

Use Amplitude:get_experiments with experiment ID to capture:

• Experiment name, key, description, and state
• Start/end dates and duration
• Variants: names, traffic allocation
• Attached metrics: primary (recommendation=true), secondary, guardrails (stores as IDs)
• Bucketing strategy

Get metric names:

• Extract metric IDs from the experiment response (e.g., "c4pn8fkv")
• CRITICAL: Amplitude MCP cannot retrieve metric names by ID directly
• Workaround options:
  1. Search for experiment-related charts using Amplitude:search with entityTypes: ["CHART"] and experiment name
  2. Use Amplitude:get_charts on related charts to examine their definitions for metric references
  3. Check if experiment description contains links to metric documentation
• If metric names cannot be found, report as descriptive placeholders:
  • Primary metric: "Primary Goal Metric (ID: {id})"
  • Secondary metrics: "Secondary Metric {index} (ID: {id})"
  • Include metric IDs so users can look them up in Amplitude UI