spark-optimization

Spark performance optimization

Most Spark tuning advice on the internet is a list of settings. That is the wrong shape: the same setting helps one job and hurts the next. What actually works is finding the one stage that dominates the runtime, learning what it is waiting on, and changing that.

The History Server has the evidence to do this properly — per-task metric distributions, per-node SQL plan counters, executor timelines. This skill is about extracting it in a fixed order so the recommendation is derived rather than guessed.

The rule that matters most

Quantify before recommending. Every recommendation should carry the number that justifies it and, where possible, the expected size of the win:

Stage 14 spilled 8.2 GB to disk across 200 tasks and accounts for 62% of wall-clock time. Raising spark.sql.shuffle.partitions from 200 to 800 puts each partition near 128 MB and should eliminate the spill.

Not:

Consider increasing shuffle partitions and executor memory.

A recommendation without a measurement is a guess, and engineers can tell.

Workflow

1. Frame the run

list_applications(app_id="<id>")
get_executor_summary(app_id="<id>")

Get the total duration and the executor totals: task count, GC time, input bytes, shuffle read/write. This is your denominator — everything later is a share of it.

Two derived numbers to compute immediately:

• GC ratio = total_gc_time / total_duration. Above ~0.10 means memory pressure is a first-order problem and will distort everything else.
• Shuffle-to-input ratio = total_shuffle_write / total_input_bytes. Well above 1 means the query moves more data than it reads — usually a join or aggregation strategy problem, and the highest-leverage thing to fix.

2. Find where the time actually goes

get_job_bottlenecks(app_id="<id>", top_n=10)

One call that returns the slowest stages and jobs, stages with significant memory spill, the GC pressure ratio and executor utilisation. Treat its recommendations as leads, not conclusions — they are threshold-based and do not know your data.