This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 87%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
I have a pipeline in Synapse which calls a data flow. That data flow has started to fail and return the below error.
{"message":"Job failed due to reason: at Source 'RawTransaction': org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 43.0 failed 4 times, most recent failure: Lost task 8.3 in stage 43.0 (TID 3283, 58924e9c16f8411a93ee73d20b870adc0004d790991, executor 1): org.apache.spark.SparkException: Exception thrown in awaitResult: \n\tat org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)\n\tat org.apache.spark.util.ThreadUtils$.parmap(ThreadUtils.scala:290)\n\tat org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.readParquetFootersInParallel(ParquetFileFormat.scala:538)\n\tat org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$9.apply(ParquetFileFormat.scala:611)\n\tat org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$9.apply(ParquetFileFormat.scala:603)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.app. Details:at Source 'RawTransaction': org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 43.0 failed 4 times, most recent failure: Lost task 8.3 in stage 43.0 (TID 3283, 58924e9c16f8411a93ee73d20b870adc0004d790991, executor 1): org.apache.spark.SparkException: Exception thrown in awaitResult: \n\tat org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)\n\tat org.apache.spark.util.ThreadUtils$.parmap(ThreadUtils.scala:290)\n\tat org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.readParquetFootersInParallel(ParquetFileFormat.scala:538)\n\tat org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$9.apply(ParquetFileFormat.scala:611)\n\tat org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$9.apply(ParquetFileFormat.scala:603)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)\n\tat org.","failureType":"UserError","target":"Prepare_forsight_All","errorCode":"DFExecutorUserError"}
I have tried switching from memory optimised to general compute, and increasing the cores on the IR but to no resolution.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 35%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERC%3C/text%3E%3C/svg%3E)
Do I have an added job
I was able to resolve this issue. There were two corrupted files out of many hundreds that were fine in the source. Removing these two corrupted files allowed the data flow to process and complete successfully again as normal.
Hi @Steven Howe ,
Glad to know that your issue has resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.
----------------------------------------------------------------------------------------
Do click on Accept Answer and Up-Vote on the post that helps you, this can be beneficial to other community members.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 6%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUC%3C/text%3E%3C/svg%3E)
@Steven Howe I have an almost similar error. I am executing a pipeline with a Data Flow Activity that reads and writes parquet files organized in a partitioned folder structure in ADLS Gen 1. The Data Flow and Pipeline validation are fine. I can also see data in Sink Data Preview. However, on running the pipeline, I am repeatedly getting the below error
Do we know the precise reason for this error.
Could you please advise.
@Steven Howe how were you able to identify the 2 corrupted files out of many source files?
The copy activity was interrupted part way through as the source database went offline which then caused the failure to complete writing the files properly. These were easily found as they were the most recently modified files.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 84%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
this worked for me as well, thank you so much for sharing.