Spark stage high I/O

A stage might be slow because it processes more data than the cluster can read, write, or shuffle in the time available. To fix an I/O bound stage:

• Reduce the amount of data the stage processes.
• Speed up how the stage reads and writes data.
• Add compute capacity.

Use this page after you rule out skew and spill in the longest stage. The stage's I/O columns show whether the stage is I/O bound and, if so, whether the bottleneck is input, output, or shuffle.

Review the I/O columns for the longest stage:

What is high I/O?

Each CPU core can read and write about 3 MB per second. To check whether a stage's I/O is high, compare its throughput per core with that rate:

1. Find the highest value in the stage's I/O columns.
2. Divide that value by the total number of CPU cores across all workers.
3. Divide the result by the stage duration in seconds.

If the result is around 3 MB per second, the stage might be I/O bound.

High input

If the Input column is large, the stage spends a lot of time reading data. To find which table or files the stage reads, see Identify an expensive read in the SQL DAG. Then try these approaches to speed up reads:

• Store the data in Delta tables.
• Use liquid clustering to improve data skipping.
• Enable Photon, which can speed up reads, especially for wide tables.
• Make the query more selective so that it reads less data.
• Reconsider the data layout so that data skipping is more effective.
• If the stage reads the same data multiple times, use the disk cache.
• If the stage runs a join, check whether dynamic file pruning applies.
• Increase the size of the cluster or use serverless compute.

High output

If the Output column is large, the stage spends a lot of time writing data. Try these approaches:

• Reduce rewrites. To check whether the stage rewrites data, see Determine if Spark is rewriting too many files. If the stage rewrites a lot of data:
  • If the write is a merge, optimize the merge.
  • Use deletion vectors to mark existing rows as removed or changed without rewriting the Parquet file.
• Enable Photon, which can speed up writes.
• Increase the size of the cluster or use serverless compute.

High shuffle

If the Shuffle Read or Shuffle Write column is large, the stage spends a lot of time shuffling data between executors for joins or aggregations. To reduce shuffle time, Databricks recommends setting spark.sql.shuffle.partitions to auto so that Spark sets the number of shuffle partitions based on the query plan and input data size. See AQE configuration.

No high I/O

If none of the I/O columns are large, the stage is slow for another reason. To find the cause, see Slow Spark stage with little I/O.