maui: Skill for Claude Code

Review Test Failures

Classify failing CI checks and tests associated with a PR, compare them against the base branch, and decide whether the PR's CI is ready to merge. The goal is to tell the author whether failures are likely caused by the PR changes or likely unrelated (flaky tests, infrastructure, missing baselines, or failures already present on the base branch), and to summarize that into one overall merge-readiness verdict.

This is the automated, deterministic counterpart to the interactive azdo-build-investigator skill. Both reason from the same shared facts (see below); this skill additionally runs in the gh-aw runtime where the ci-analysis plugin is not available, so it relies entirely on the gathered context files.

Shared MAUI CI facts

Read .github/docs/maui-ci-facts.md for the canonical pipeline names/IDs, AzDO data sources, XHarness exit-0 blind spot, test deduplication rule, baseline-comparison rule, visual-baseline rule, platform-mismatch guidance, Gradle/CFSClean signatures, common failure patterns, and the merge-readiness criteria. Do not restate those facts from memory — they change in one place.

Inputs

Use the context produced by .github/skills/review-test-failures/scripts/Gather-TestFailureContext.ps1.

Expected context files:

• context.json — structured PR, check, build, log, baseline, and deduplicated test-failure data.
• context.md — compact human-readable summary of the same data.

Key fields to use:

• gate — deterministic merge-readiness gate computed in the gatherer (not LLM judgment). Use it as a hard ceiling, never override it upward:
  • gate.verdictCeiling — the most favorable overall verdict the evidence permits (Insufficient data / Needs human investigation / Not ready / No failures found / Ready to merge). Your overall verdict MUST NOT be more favorable than this.
  • gate.ceilingReasons[] — exact reasons (with check names) that set the ceiling.
  • Coverage counts: totalChecks, passingOrNeutralChecks, failingChecks, pendingChecks, inaccessibleFailingChecks, unmappedFailingChecks, unexplainedFailedLegs (failed build legs that produced no extractable failure — a build break with no test name, or an unreadable log; any value > 0 caps the ceiling at Needs human investigation).
  • gate.unaccountedFailingChecks (+ unaccountedFailingCheckNames[]) — failing checks backed by an accessible build that produced no extractable failure and no unexplained-leg record (the build's log threw, had no log id, or fell past the per-build failed-record cap). This is the earned-green guard: a red check we could reach but pulled zero reason from must not be read as green. Any value > 0 caps the ceiling at Needs human investigation.
  • gate.abortedFailingChecks (+ abortedFailingCheckNames[]) — failing checks whose GitHub conclusion did not finish cleanly: CANCELLED, TIMED_OUT, STARTUP_FAILURE, STALE, or ACTION_REQUIRED. A cancelled/timed-out check is red but its aborted AzDO legs can carry no error issue (so they never become unexplained legs) — e.g. a PR-induced hang that got a job cancelled. Without this guard, a dismissible sibling failure on the same build could "earn" the build green and mask the abort. An aborted check is never a trustworthy pass, so any value > 0 caps the ceiling at Needs human investigation.
  • gate.canceledBuildChecks (+ canceledBuildCheckNames[]) — checks backed by an AzDO build whose own metadata result is canceled, regardless of how the GitHub check conclusion reads. This is broader than abortedFailingChecks (which keys only on the GitHub conclusion): a build can be canceled mid-flight while a leg had already posted FAILURE or even SUCCESS, so the canceled build slips past the conclusion-based guard. A canceled build's legs frequently carry no error issue, so a dismissible sibling can falsely "account" for it. A canceled build is never a trustworthy pass, so any value > 0 caps the ceiling at Needs human investigation.
  • gate.deviceTestUnverified (+ deviceTestUnverifiedNames[]) — device-test checks (maui-pr-devicetests) that read GREEN but whose Failed == 0 could not be positively confirmed. XHarness exits 0 even when device tests fail, so a green device-test check is not evidence of a clean run. The gatherer force-inspects every device-test build and only clears it when a fail count was positively observed and was all-zero (Helix aggregated, or the authenticated test-API when a token is present) over a COMPLETE, error-free read — the Helix path requires every discovered job's aggregate to be read without a thrown error, and the test-API path pages through all test runs and refuses confirmation if the run set was truncated (a failing run could sit in the unread tail). When no such confirmation is available — the common case in the gh-aw runner, which has no AzDO token — the green device-test check is unverified and caps the ceiling at Needs human investigation. A SKIPPED device-test check does not cap (tests did not run); a RED one is handled as an ordinary failing check.
  • gate.legsRegressedVsBase (+ legsRegressedVsBaseNames[]) — distinct failures that are red on the PR but GREEN on the same leg across several recent completed base builds and red on none of them (a deterministic, computed job-level regression). Any value > 0 caps the ceiling at Not ready — a Ready to merge / No failures found verdict is then forbidden. Sampling several base builds (not one) is what separates a real regression from a base-branch flake that merely happened to pass its one sampled base run — except a deterministic build break (crossgen/NativeAOT/linker/ MSBuild, which compiles or it doesn't), where a single green base build is proof enough. This is the comparison that catches build-job breaks (crossgen/R2R, NativeAOT) the test-level baseline cannot. A device-test BUILD break (source = azdo-build-error) IS counted here because it is deterministic; only device-test TEST results are excluded (XHarness exit-0 blind spot) — they are surfaced but never hard-capped.
  • gate.unattributedFailures (+ unattributedFailureNames[]) — distinct failures the deterministic prior could attribute neither way: not a clean regression vs base, not pre-existing on base, not a known issue (deterministicAttribution = indeterminate). Causes: the leg was flaky on base (red on some sampled base builds, green on others → flaky-on-base), the leg was green on base but on too few samples to confirm a regression (succeeded-on-base-unconfirmed), the base build was missing/unreadable, or a device-test TEST result outside the build-error class. They are neither provably PR-caused nor dismissible as pre-existing/known, so any value > 0 caps the ceiling at Needs human investigation.
  • Evidence counts: failuresAlsoOnBaseline, failuresMatchingKnownIssue, failuresRetriedStillFailing, baselineInconclusiveRows.
• failures.unique[] — distinct PR failures (deduped by test name + OS platform). This includes build-job breaks (crossgen/R2R, NativeAOT/ILC, linker, MSBuild error, and fatal non-coded breaks — native crash/segfault/OOM, test-host crash, unhandled exception), which carry a synthetic name like Build macOS (Debug) - Failed to load assembly and a source of azdo-build-error — they are real failures, not noise. Each carries:
  • alsoFailsOnBaseline (true when the same test+platform also fails on the most recent base-branch build — scoped to the same pipeline definition, so a failure in one pipeline is never dismissed by a same-named failure that only occurred in another),
  • legBaselineResult / legRegressedVsBase / legAlsoFailsOnBase — the computed job-level baseline diff for this failure's leg, computed over the last few completed base builds (not a single base build) so a base-branch flake is not mistaken for a regression: succeeded-on-base + legRegressedVsBase = true means the SAME leg was GREEN across several recent base builds and red on none of them, and is now red on the PR (strongest PR-caused signal); failed-on-base + legAlsoFailsOnBase = true means the same leg was already red on at least one sampled base build — but note this is only leg-level corroboration, NOT proof that this specific test is pre-existing (the leg can fail on base at a different test), so on its own it does not dismiss the failure; flaky-on-base means the leg was red on some sampled base builds and green on others (demonstrably flaky on base) — indeterminate, never a regression; succeeded-on-base-unconfirmed means the leg was green on base but on too few samples (fewer than MinBaseGreenSamples, e.g. only one readable base build) to rule out flakiness — indeterminate, not a confident regression; absent-on-base means the leg name did not exist on the sampled base builds (indeterminate — do not treat as a regression). The computed regressed-vs-base set is pre-filtered to stay trustworthy: a provisioning/infrastructure failure (Android SDK Failed to find package, avdmanager, disk-full — environmental and nondeterministic) and any failure that was flaky on base in another leg are both held to legRegressedVsBase = false so they fall to indeterminate rather than masquerading as a deterministic regression. Each failure also carries baseSampleCount / baseGreenCount / baseFailedCount (how many recent base builds were read, and on how many the leg was green vs red) as regression-confidence evidence,
  • deterministicAttribution — a computed prior you MUST start from, one of regressed-vs-base (treat as Likely PR-caused unless you can cite why the base comparison is invalid, e.g. a known-flaky base leg), pre-existing-on-base (treat as Likely unrelated — the exact test+platform is also red on base, the only signal strong enough to dismiss), known-issue (the exact test+platform is also red on base and the failure message matches a known issue — a richer label for the same dismissable, not-PR-caused case), or indeterminate (everything else: a leg-only base match, an uncorroborated known-issue text match (a known-issue regex hit on a test that did not exact-match base — a leg being red on base at a different test is no longer treated as corroboration), a base/PR reason conflict (see baselineReasonConflict), an ambiguous/missing base, or a genuinely unknown failure — NOT dismissable, caps the ceiling at Needs human investigation). You may override regressed-vs-base/pre-existing-on-base only with an explicit, cited reason,
  • matchesKnownIssue ({number,title,url} when the failure message matches an open Known Build Error issue; null otherwise) — a documented-flake hint. A text match alone is NOT enough to dismiss a red check (a broad matcher can shadow a real PR break): it only becomes the dismissable deterministicAttribution = known-issue when the exact same test+platform also failed on the base build (leg-level corroboration is too coarse and no longer dismisses). Cite the issue number, but defer to the computed attribution,
  • matchesCiScan ({number,title,url,class,branch,occurrences,matchKind} when this failure is documented in the repo's open [ci-scan] registry for the PR's base branch family; null otherwise) — the [ci-scan] issues are the MAUI CI Failure Scanner (an agentic ci-status-* workflow) tracking recurring flakes, regressions, and build breaks on the main / net11.0 base branches across many builds — i.e. multi-build base-branch history, strictly broader than the few recent base builds the leg diff samples. It is used in one direction only: when the leg diff computed a few-build regressed-vs-base and ci-scan documents that exact test (matchKind=test) or its whole leg (matchKind=leg, only for OneTimeSetUp/mass/env/build-break leg-wide issues) as failing on the base branch, the regression is demoted to indeterminate (NHI) and ciScanDemoted=true is set. This is a false-RED reduction only — a ci-scan hit can NEVER turn a red check green (it is an LLM-generated, possibly-stale hint, so it is never a dismissal-to-green signal; it only moves an over-confident Not ready down to Needs human investigation). Branch family must match (a main PR is never demoted by a net11.0 ci-scan issue). Surface the linked issue + occurrence count for the human,
  • retriedStillFailing (true when CI retried the leg and it still failed — this is evidence the failure is persistent, NOT a one-off flake).
  • baselineReasonConflict (true when this failure exact-matches a base failure by name+platform but the two fail for different reasons — e.g. a PR-introduced NullReferenceException vs a base-branch TimeoutException in the same test). The name-based dedup key is message-blind for test failures, so without this a PR-caused break could be laundered as pre-existing-on-base. When set, the dismissal is refused and the attribution is forced to indeterminate. It fires when both reasons are known and differ (wrapper exceptions like AggregateException are unwrapped to the inner cause — and when a wrapper carries multiple inner exceptions they are collapsed into a sorted compound token, so a PR-introduced inner cannot hide behind a base-matching first inner), and — for test failures where neither side yields a known reason — as a fallback when the PR's normalized message fingerprint is absent from base for that test (the fingerprint preserves identifier-internal digits and hashes any long tail, so two distinct breaks that differ only by an identifier digit or a far-out-of-line suffix stay distinct). These paths fire only on data present on both sides; the one exception is a dismissible test failure that exposes no reason token and no message text at all (e.g. a device/UI result with an empty errorMessage) — that offers zero corroboration that it is the same failure as the name-matched base failure, so it is also forced to indeterminate rather than laundered as pre-existing. A noisy or partially-present message still never inflates false reds.
  • scopeGuardTripped (true when a pre-existing-on-base or known-issue dismissal was refused because the PR actually edits the test file behind the failure). When the PR touches the very test that is failing, a base or known-issue text match is no longer safe grounds to dismiss it — the PR may have changed the test so it now fails for a new reason that merely coincides with the base/known text. The attribution is forced to indeterminate (which caps the ceiling at Needs human investigation) instead of being laundered green.
• failures.baseline[] — distinct failures extracted from the base-branch build(s).
• failures.baselineMatchCount — how many distinct PR failures also fail on the base.
• visualEvidence — public AzDO result/attachment metadata for visual snapshot failures. It is supplementary evidence and never changes the deterministic gate.
• visualAssets — present when the trusted publisher produced durable GitHub-hosted images. A deterministic merger inserts a bounded subset of these comparisons into the single final analysis comment.
• knownIssues — {queried, matcherCount, error}. If queried is false (gh failed), the absence of a matchesKnownIssue hit proves nothing — say so.
• baselineSummary[] — which base build was inspected per pipeline definition, its result, and how many baseline failures were found (a succeeded base build is noted as strong evidence that matching failures are not pre-existing).
• checks.interesting[], builds[], scope.* — failing checks, AzDO build evidence, and the PR's changed-file/platform/area scope.