Payload Experimental Reverts

Payload Experimental Reverts

This skill experimentally tests medium-confidence candidate PRs by opening draft revert PRs, triggering payload jobs, and evaluating results. It operates in two phases separated by a CI wait period. All state is tracked in the payload results YAML file via the payload-results-yaml skill — no separate tracking file is created.

When to Use This Skill

Use this skill when the /ci:payload-experiment command identifies candidate PRs with medium confidence (score 60-84) that cannot be conclusively attributed to a failure through static analysis alone. The experiment creates real tests to determine causality.

Inputs (passed in-context by the caller):

• results_yaml_path: Path to the payload results YAML file (e.g., ./payload-results-{tag}.yaml)
• candidates: List of medium-confidence PRs to test experimentally, each with:
  • pr_url, pr_number, component, title, confidence_score
  • failing_jobs: List of {job_name, prow_url, is_aggregated, underlying_job_name}

Required Skills

Before starting, you MUST load the following skills (they define output schemas used when updating results):

1. payload-results-yaml — schema for the payload results YAML file
2. payload-autodl-json — schema for the autodl JSON data file

Prerequisites

1. GitHub CLI (gh): Installed and authenticated
2. JIRA MCP: Configured for creating TRT issues (needed in Phase 2 for confirmed causes)
3. Repository Access: User must have push access to their fork of each target repository

Implementation Steps

Phase 1: Set Up Experiments

For each medium-confidence candidate, launch a parallel subagent (do NOT set the model parameter):

1.1: Check for Merge Conflicts

Before opening a revert PR, preemptively check whether the revert will have merge conflicts:

# Clone the repo (shallow for speed)
git clone -b <base_branch> --depth 50 "https://github.com/<org>/<repo>.git" /tmp/experiment-check-<pr_number>
cd /tmp/experiment-check-<pr_number>

# Attempt the revert without committing
git revert -m1 --no-commit <merge_sha>

# Check for conflicts
git status --porcelain