triage-ci-flake

Triage CI Failure

Overview

Systematic workflow designed specifically for triaging and fixing end-to-end (e2e) test failures in CI, especially flaky Playwright tests that pass locally but fail in CI. E2E tests that made it to main are usually flaky due to timing, bundling, or environment differences.

CRITICAL RULE: You MUST run the reproduction workflow before proposing any fixes. No exceptions.

When to Use

• CI test fails on main branch after PR was merged
• Test passes locally but fails in CI
• Test failure labeled as "flaky" or intermittent
• E2E test timing out in CI only
• User provides a PR URL/number with failing checks

PR-Based Workflow (When PR URL/Number Provided)

When the user provides a PR URL (e.g., https://github.com/payloadcms/payload/pull/16701) or PR number:

Step 1: Fetch PR Status Checks

First, use tool_search with query "github pull request status checks" to load the GitHub tools.

Then use the github-pull-request_pullRequestStatusChecks tool to get all failing checks:

Tool: github-pull-request_pullRequestStatusChecks
Parameters:
  pullRequestNumber: <extracted from URL or provided>
  repo: { owner: "payloadcms", name: "payload" }

Step 2: Aggregate Failing Tests

Parse the status checks response and create a summary table:

Suite	Check Status	Target URL
admine2elist-view	failed	[link]
plugin-import-export	failed	[link]

For each failing check, the context field contains the test suite name (e.g., admin__e2e__list-view).

Step 3: Extract Failure Details from Each Check

For each failing check:

1. Visit the targetUrl to get detailed failure logs
2. Extract the specific test name and error message
3. Add to the aggregated failure list

Present a consolidated summary:

## Failing Tests Summary

### 1. admin**e2e**list-view (3/4)

- **Test**: "should use custom pagination limit"
- **Error**: Locator `.per-page__button` not found
- **File**: test/admin/e2e/list-view/e2e.spec.ts:1495

### 2. plugin-import-export

- **Test**: "should inherit limit from list view URL"
- **Error**: Locator `.per-page__button` not found
- **File**: test/plugin-import-export/e2e.spec.ts:150