spark-rca

Spark failure root-cause analysis

A failed Spark application leaves a complete forensic record in the event log. The History Server exposes it; the job here is to read it in the right order and stop at the first piece of evidence that actually explains the failure, rather than guessing from the driver's top-level error message.

That top-level message is almost always a symptom, not a cause. Job aborted due to stage failure tells you nothing. The real cause sits in a task exception, four levels down. Getting to it reliably is what this skill is for.

The rule that matters most

Never name a cause you have not seen in the data. "Probably a memory issue" is worse than useless — it sends an engineer to change executor.memory on a job that actually failed on a bad cast. Every claim in the final answer must point at a specific stage, task, exception, metric or config value that a tool returned. If the evidence runs out before the cause is clear, say exactly what is missing and what would reveal it.

Workflow

Work through these in order. Stop early only when the cause is unambiguous — and "unambiguous" means you have the exception text or a metric that forces the conclusion, not a plausible story.

1. Establish the application

list_applications(app_id="<id>")

Note the status, duration and — important — the attempts array. A YARN application with several attempts means the whole application was retried, which usually indicates an AM/driver-level failure rather than a task failure. If there is more than one attempt, pass app_attempt_id to the later tools so you are reading one specific run rather than an arbitrary one.

If you only have a name, a time window, or a cluster, find the id first: list_applications(status=["completed"], min_date="2026-08-01").

2. Find the failed unit of work

list_jobs(app_id="<id>", status=["FAILED"])
list_stages(app_id="<id>", status=["FAILED"])

A subtlety worth knowing: a Spark application can fail with no job marked FAILED. The driver may have died before the failure was recorded, or the job may have been killed externally. When the status filters come back empty, list without a filter and look for non-zero numFailedTasks / numFailedStages, or for a job that never completed (completionTime is null). An application whose last job is incomplete usually died in the driver — jump to step 5.