spark

Spark

Purpose

Write Spark jobs whose cost is understood. Almost all Spark performance problems are one of three things: too much shuffle, skewed partitions, or reading far more data than the query needs.

When to Use

• Building or reviewing a Spark pipeline.
• A job that is slow, failing with out-of-memory errors, or has one straggling task.
• Tuning partitioning and join strategy.
• Reading the Spark UI to diagnose a stage.

Capabilities

• Partitioning strategy and repartitioning.
• Shuffle minimization and broadcast joins.
• Skew detection and mitigation.
• Caching and persistence levels.
• File-format and predicate-pushdown optimization.
• Spark UI interpretation.

Inputs

• The job, its input data volume, and its physical plan.
• The Spark UI: stage timings, task distribution, shuffle read/write.
• Cluster resources.

Outputs

• A plan with fewer or smaller shuffles.
• Balanced partitions with no straggling tasks.
• Measured improvement in wall-clock time and cost.

Workflow

1. Read the plan first — df.explain(True). Every Exchange is a shuffle, and a shuffle writes to disk and crosses the network. It is the dominant cost.
2. Prune early — Select the columns and filter the rows you need before joining, not after. With Parquet, this pushes down to the file reader and never reads the data at all.
3. Broadcast the small side — A join where one side fits in memory (roughly under 100 MB) should be a broadcast join. That eliminates the shuffle entirely.
4. Find the skew — In the Spark UI, look at the task duration distribution within a stage. If the max is 50x the median, one partition holds most of the data. That single task is your job's runtime.
5. Mitigate the skew — Salting the key, or enabling adaptive query execution's skew join handling.
6. Cache only what is reused — Caching a DataFrame used once costs memory and gains nothing.

Best Practices

• Enable adaptive query execution (spark.sql.adaptive.enabled=true). It coalesces partitions, converts joins to broadcasts at runtime, and handles skew automatically. It is on by default from Spark 3.2 and is the single largest free improvement available.
• A shuffle is the most expensive operation in Spark. groupBy, join, distinct, and repartition all shuffle. Count them in the plan.
• collect() brings the entire dataset to the driver. On anything non-trivial this is an out-of-memory error waiting for a larger input.
• Use Parquet or Delta, never CSV or JSON, for anything that will be read more than once. Columnar formats support predicate and projection pushdown; row formats do not.
• Too many small partitions means task-scheduling overhead dominates; too few means poor parallelism and memory pressure. Aim for partitions of roughly 128 MB.
• cache() is lazy. It does nothing until an action runs, and it can silently fall back to recomputation if it does not fit in memory.