debug-failing-ci

Debug a failing CI pipeline

A red pipeline measures a difference: between your machine and the runner, between two commits, or between two runs of the same commit. Find the difference, fix what it reveals, and never make CI green by making it look away.

How this works in Gaffer. You never push and cannot re-run CI yourself. The runner pushes your branch and opens a PR only when the operator has enabled that, and only then can its opt-in CI gate run: a red check sends the delivery back to rework with the failing check attached as evidence. So your proof is a local reproduction under CI's exact conditions — fails before, passes after — and any later CI run is the runner's.

Steps

1. Get the real failure text. Read get_ticket (the description, last_review_feedback and latest_events) plus any review-feedback block in your prompt. Find the FIRST error in the log, not the last — later failures are usually consequences. Note the step, the exact command, and the exit code. Call search_lore for known CI gotchas in this repo. If only a check name and URL are available and you cannot open the log (no network or credentials), go on to step 3 anyway: the workflow file tells you what CI ran.
2. Classify before fixing. Each class has a different fix:
   • code / test — the change is wrong, or a test encodes stale behaviour;
   • flake — passes and fails on the same commit (order, time, shared state, network);
   • environment drift — tool or runtime version, OS (BSD vs GNU sed/date), locale or timezone, a file present locally but ignored or untracked in git;
   • workflow config — YAML syntax, matrix, needs:, paths filters, a secret that is absent on forks;
   • infrastructure — runner outage, registry or rate-limit error, disk full. Do not change code for this: report it (see "Stop and escalate when").
3. Reproduce CI's conditions. Open the workflow (.github/workflows/*.yml, .gitlab-ci.yml, …) and copy the exact commands, the runtime versions (setup-node, .nvmrc, python-version), the env vars and the working directory. Run the build, test and lint commands, not your local alias. Never run the workflow's install or setup steps (npm ci, pnpm install, pip install, bundle install, uv sync, playwright install, …): the worktree uses the existing install. If the failure is in an install step, reason from the lockfile and escalate. Compare node --version (or the ecosystem's equivalent) with the pinned version. List what your run depends on that a fresh checkout lacks: git status --porcelain --ignored shows untracked and ignored files (generated code, a local config, a cache). Most "works on my machine" failures die here.
4. Bisect when the cause is not obvious. If the job passed on an earlier commit: git bisect start <first-red> <last-green>, then git bisect run <failing command>, then git bisect reset. For a suspected flake, measure the rate with a loop (for i in {1..20}; do <cmd> || echo FAIL; done) and run the test alone vs with the suite to expose order dependence.
5. Fix the cause. A real test failure → fix the code (the fix-bug skill). A flake → fix the nondeterminism (the fix-flaky-test skill). Drift → pin the version, install the tool in the workflow, or make the script portable (the shell-scripting skill). A workflow error → fix the YAML and keep permissions: minimal.
6. Prove it locally. Show the CI command failing before your fix and passing after; for a flake, the loop at 20/20. Record it via the record-evidence skill as test_output: the log excerpt, the cause in one sentence, the passing command and its summary. Use ci_run only for a real CI run URL you actually observed. On a resume, where that call is refused, put this in your final message instead (see record-evidence).
7. Leave the pipeline better, briefly. If the log hid the cause, add the one missing diagnostic (print the tool version, upload the test report) when it is small.