Long Spark stage with one task

A long-running stage with only one task has no parallelism. While the task runs, one CPU does the work, and the rest of the cluster might be idle. Adding workers doesn't help, because Spark can't spread the work across them.

Use this page when the longest stage has exactly one task. If the stage has many tasks, check it for skew and spill. See Skew and spill.

Common causes and fixes

A single-task stage most often has one of these causes. Find the one that matches your code, and then apply the fix:

• An expensive UDF on small data. Small input might land in a single partition, so one task runs the whole UDF. Rewrite the UDF using native functions if you can. Otherwise, repartition() the DataFrame before you apply the UDF. See Slow UDFs.
• A window function without PARTITION BY. Without a PARTITION BY clause, the window spans the whole dataset, so one task processes every row. Add a PARTITION BY clause if your logic allows it.
• An unsplittable file type. Spark can't read files such as gzip files in multiple parts, so one large task reads each file. If you read the same files repeatedly, ingest them one time into a Delta Lake table and query the table instead.
• The multiLine option on JSON or CSV reads. Set multiLine only when records span multiple lines. See Read multi-line JSON files.
• Schema inference on a large file. Specify the schema instead of inferring it. See Specify a schema.
• repartition(1) or coalesce(1). These force all data into one partition. Remove them, or use more partitions, unless you need a single output file.