debugging-ci-failures

Debugging PostHog CI failures

Before you propose a change to CI, check things already tried for the idea. It records what was measured, and why some good-sounding changes were reverted or rejected.

Find the first meaningful failure, classify it, reproduce the smallest useful case locally when appropriate, and report the result. Avoid public-visible or irreversible actions unless the user explicitly asks.

Always start with the hogli ci:insights digest. It aggregates across runs and branches, which gh cannot do cheaply, and tells you whether a failure is likely trunk-borne, gate-only, or isolated to a small set of branches. gh is authoritative for one run's current state and attribution. Use the digest to decide what to inspect; use gh to confirm whose failure it is and exactly what failed in a given run.

This skill triages and classifies. Once a failure is confirmed flaky, hand off to the fixing-flaky-tests skill, which owns local reproduction, root-cause fixing, and N-run validation. For "who broke master" — the culprit commit and the commit that fixed it — hand off to the investigating-ci-failures skill, which owns the green/red boundary analysis. For aggregate pipeline health (is CI getting slower, which workflow is the long pole, how long PRs take to merge), read diagnosing-ci-and-merge-bottlenecks. Both are product skills under products/engineering_analytics/skills/, not invocable here: read the SKILL.md at that path.

Rule out a platform outage first

GitHub Actions goes down often enough that it belongs before any log reading, and a platform incident makes every other signal a symptom. It costs two page loads: https://www.githubstatus.com/ for GitHub, https://status.depot.dev/ for the runners. Check them whenever failures are broad — several workflows at once, a burst of runs failing together, jobs dying before Checkout, or anything red across unrelated PRs.

Report an outage as an outage, name the component, and stop recommending reruns.