vscode: Skill for Claude Code

Diagnosing Flaky Smoke Tests

Use this skill for failures from the Azure DevOps Flaky Smoke Tests pipeline:

• Organization: https://dev.azure.com/monacotools
• Project: Monaco
• Definition ID: 700
• Pipeline YAML: build/azure-pipelines/product-smoke-flaky.yml
• Scheduled runs: twice daily on main

The pipeline builds VS Code from source and runs the complete Electron smoke suite once per entry in the iterations parameter. Each platform runs its iterations sequentially in one job. The exception is Windows unit and integration tests, which are each split into two parallel jobs: WindowsSmokeFlakyUnit_odd / WindowsSmokeFlakyIntegration_odd run the odd iterations and WindowsSmokeFlakyUnit_even / WindowsSmokeFlakyIntegration_even run the even ones. Each shard has its own artifacts (logs-windows-x64-<unit|integration>-<odd|even>-<attempt>) and its own Slack tally line.

Important Pipeline Behavior

• Each iteration is a separate timeline task named Smoke test iteration <N>/<total> (Electron).
• Iteration tasks use continueOnError: true, so a failed iteration normally has result succeededWithIssues and later iterations still run.
• Do not infer smoke-test health from the overall build or job result. Inspect every iteration task.
• Logs are published once per platform job from the shared .build/logs directory. The artifact is job-scoped, not iteration-scoped: smoke-test-runner.log can contain multiple iterations, and suite directories are not separated by iteration.
• Do not cancel a platform job while investigating if you still need its logs artifact. Cancellation can skip the publish steps entirely, even when one or more failed iteration task logs are already available. If a validation run has become predictably red, either let the current job finish or queue a single diagnostic iteration and let that job publish before iterating.
• Later iterations can overwrite or truncate files in the shared logs directory. The final artifact may therefore contain the last iteration's suite directory or runner log but not the failed iteration's detailed diagnostics. Preserve the exact task log, and use a one-iteration diagnostic run when extension-host, renderer, mock-server, screenshot, or trace evidence from the failed attempt is required.
• The exact iteration task log is authoritative for its failure summary and time range. Correlate that range with the job-level artifact.
• Iterations can see persisted smoke-test state from earlier iterations, such as historical session rows. Never assume the newest-looking row or a matching prompt belongs to the active conversation without verifying the active view.