debug-failed-run

Debug a failed workflow run

Start from the run id and work outward. Every step below is read-only until the final decision, so none of it costs GPU time or destroys the evidence you need.

0. Find the run id

npa workbench workflow list --project <alias> --limit 50 --json

Runs are durable S3 objects, so this works after the controller, the cluster, or your shell is gone. If you only have an S3 prefix, every command below accepts an s3:// run prefix wherever it accepts a run id.

1. Status first — it names the failure class

npa workbench workflow status <run-id> --project <alias> --json

This is the single most informative command, because for a PENDING job it reports the pod-level reason rather than just "pending". That distinction is the whole triage:

• PENDING + ImagePullBackOff → registry problem, go to step 4. The job will never fail on its own; Kubernetes retries pulls forever.
• PENDING + Unschedulable → capacity or accelerator-name problem, step 5.
• FAILED → the payload ran and exited non-zero, step 3.
• FAILED_STARTUP → the controller itself failed identically several times (threshold --startup-failure-threshold, default 3). This is infrastructure, not your payload.

Runtime-supervised runs also expose supervisor.classification and supervisor.recovery in JSON status. actionable_configuration means automatic retry has stopped and only the exact recorded attempt was cancelled; transient_infrastructure records adoption or a new attempt under the same run ID until the finite --max-infrastructure-recoveries policy is exhausted; payload uses only explicit --retries; unknown blocks relaunch to prevent duplicates. INFRASTRUCTURE_RECOVERY_EXHAUSTED is terminal durable evidence, not permission to increase payload retries implicitly.

Add --watch --interval 10 to follow a run to a terminal state. Use --cached only when the live controller is unreachable: its output is explicitly marked CACHED and is not automation-trustworthy — never gate a decision on it.