Diagnosing a long job in Spark

When one job dominates a workload's runtime, the time is usually concentrated in one or a few of that job's stages. Finding the longest stage, and reading its I/O and task count, narrows the problem down to a specific cause such as spill, skew, high I/O, or too little parallelism.

Use this page after the jobs timeline shows one or a few long jobs, or after you sort jobs by duration and open the longest one. If the timeline shows a different pattern, see the page for that pattern:

• Failing jobs or executors removed
• Gaps between Spark jobs
• Many small Spark jobs

Find the longest stage

Start by identifying the longest stage of the job. Scroll to the list of stages at the bottom of the job page and order them by duration:

Stage I/O details

Review high-level I/O data for this stage using the Input, Output, Shuffle Read, and Shuffle Write columns:

Each column shows:

• Input: How much data this stage read from storage, such as Delta Lake, Parquet, or CSV files.
• Output: How much data this stage wrote to storage, such as Delta Lake, Parquet, or CSV files.
• Shuffle Read: How much shuffle data this stage read.
• Shuffle Write: How much shuffle data this stage wrote.

Record these numbers for later steps.

Number of tasks

The number of tasks in the long stage can help identify the cause. The following screenshot shows where to find the number of tasks:

A single task means the stage runs with no parallelism, so only one core does the work while the rest of the cluster waits. If the stage has one task, see Long Spark stage with one task.

View more stage details

If the stage has more than one task, investigate further. Click the link in the stage's description to view details about the longest stage:

Next, see Skew and spill.