monitor-experiments

Experiment Monitor & Report Generator

Scan active and recently completed experiments, surface what needs attention, and report on the ones that matter.

This is a monitoring skill — keep output accessible to non-experts. Avoid statistical jargon (no p-values, no power analysis). For deep-dive analysis of a specific experiment, use the analyze-experiments skill instead.

---

CRITICAL: Managing Response Sizes

1. get_experiments: 3-5 IDs max per call. Filter using search results BEFORE fetching.
2. query_experiment responses are large. Extract only summary objects and validity flags. Ignore timeseries, xValues, bulk arrays.
3. Metric name resolution: search does NOT match metric IDs in queries. Search with entityTypes: ["METRIC"], empty queries, limitPerQuery: 50, scoped to project. Match IDs from results.

---

Report Structure

The report has two parts:

1. Summary & Actions (top) — One table + action items. Someone should be able to read just this and know the full picture.
2. Details (bottom) — Deep-dives on experiments that need attention, briefs on recently decided, one-liners for monitoring experiments, and a needs-setup list.

Do NOT duplicate information between the summary table and the details. The summary table is the single source of truth for the portfolio state. Details expand on specific experiments.

---

Instructions

Step 1: Context & Discovery

1. Call Amplitude:get_amplitude_context. If multiple projects, ask which to monitor.
2. Search for experiments:

Amplitude:search({
  entityTypes: ["EXPERIMENT"],
  appIds: [projectId],
  queries: [],
  sortOrder: "lastModified",
  sortDirection: "DESC",
  limitPerQuery: 50
})

1. Filtering rules — include experiments that are:
   • Running and not stale: Any experiment in a running state that is NOT marked as stale. Stale experiments have gone idle and should be excluded.
   • Recently decided: Completed experiments that have a decision recorded AND were modified within the last 14 days. These are worth reviewing to confirm the decision or share learnings.
2. Exclude: Drafts, disabled experiments, and stale experiments.