auditing-experiments-flags

Auditing experiments and feature flags

This skill teaches you how to run configuration audits on experiments and feature flags. All checks use the experiment and feature flag read tools (experiment-get, experiment-list, feature-flag-get-definition, feature-flag-get-all) — no SQL queries are needed for Phase 1 checks.

Usage modes

Quick check (single entity)

When the user asks about a specific experiment or flag:

1. Fetch the entity via experiment-get (experiment ID) or feature-flag-get-definition (numeric flag ID).
2. Apply the relevant checks from experiment checks or flag checks.
3. Report findings inline as markdown, grouped by severity (CRITICAL first, then WARNING, then INFO).
4. Include entity links as [Experiment: name](/experiments/id) or [Flag: key](/feature_flags/id).

Scoped audit (one domain)

When the user asks to audit all experiments or all flags:

1. Bulk-fetch via experiment-list or feature-flag-get-all.
2. Run all checks for that domain against each entity.
3. Group findings by severity, then by entity.
4. Report as inline markdown.

Full audit (comprehensive)

When the user asks for a comprehensive audit of both experiments and flags:

1. Fetch all experiments via experiment-list and all flags via feature-flag-get-all.
2. Run all experiment checks and all flag checks.
3. Apply recurring patterns to identify patterns across multiple findings.
4. If there are more than 5 entities with findings, write them to a notebook for easier navigation. Otherwise report inline. Create the notebook from the project's own notebook tools. Run search notebooks?- to load them and read the titles.

Output format

For each finding, include:

• Severity badge: 🔴 CRITICAL, 🟡 WARNING, or 🔵 INFO
• Check name: Which check produced this finding
• Entity link: Markdown link to the entity
• What's wrong: One-sentence description
• Action: What to do about it (see remediation actions)