Azure SQL Database
An Azure relational database service.
6,338 questions
Azure Pipeline Load Failure: Failed to execute dataflow with internal server error
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 31%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERW%3C/text%3E%3C/svg%3E)
Ryan Wright
0
Reputation points
Have been receiving this error for several days, attempt to re-run manually yields same result. Any guidance/insight on how to resolve?
Operation on target rptSalesDailyLoad failed: Operation on target Fact Sales Load failed: Operation on target df_factSales failed: {"StatusCode":"DF-Executor-InternalServerError","Message":"Job failed due to reason: at Sink 'FinalFactDel': Failed to execute dataflow with internal server error, please retry later. If issue persists, please contact Microsoft support for further assistance","Details":"org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 100.0 failed 1 times, most recent failure: Lost task 11.0 in stage 100.0 (TID 4626) (vm-14295733 executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container from a bad node: container_1692899079555_0001_01_000002 on host: vm-14295733. Exit status: 143. Diagnostics: [2023-08-24 18:40:18.981]Container killed on request. Exit code is 143\n[2023-08-24 18:40:18.982]Container exited with a non-zero exit code 143. \n[2023-08-24 18:40:18.997]Killed by external signal\n.\nDriver stacktrace:\n\tat com.microsoft.dataflow.FileStoreExceptionHandler$.extractRootCause(FileStoreExceptionHandler.scala:36)\n\tat com.microsoft.dataflow.transformers.DefaultFileWriter.$anonfun$write$16(FileStore.scala:1206)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n\tat scala.util.Try$.apply(Try.scala:213)\n\tat com.microsoft.dataflow.transformers.DefaultFileWriter.$anonfun$write$15(FileStore.scal"}
Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache 20,366 Reputation points Moderator2023-08-28T17:13:03.7033333+00:00 Hello @Ryan Wright,
As the error message suggests, you can retry the operation late! Did you re-try the operation?
Is this consistent error?
-
Ryan Wright 0 Reputation points
2023-08-28T17:17:25.5366667+00:00 Hi Satish,
The same error occurred after trying to kick-off manual run. This has been consistent for a couple of weeks now. Thanks, Ryan
-
QuantumCache 20,366 Reputation points Moderator
2023-08-28T20:36:19.6166667+00:00 Hello @Ryan Wright Is the Data Flow failing to run for any other new pipelines? Did you try any other new Data Flow creation and check?
Could you please share more info like the ADF Region? any screenshots of the error and the Dataflow sample? that would be helpful for me, before escalating this issue...
-
Ryan Wright 0 Reputation points
2023-08-28T20:48:12.94+00:00 This is a data pipeline that is set to run every day. It has failed repeatedly at the same stage for several weeks now. We have not created any new pipelines in months at this point.
Location: West US
This is the highest level load failure message:
Operation on target rptSalesDailyLoad failed: Operation on target Fact Sales Load failed: Operation on target df_factSales failed: {"StatusCode":"DF-Executor-InternalServerError","Message":"Job failed due to reason: at Sink 'FinalFactDel': Failed to execute dataflow with internal server error, please retry later. If issue persists, please contact Microsoft support for further assistance","Details":"org.apache.spark.SparkException: Job aborted due to stage failure: Task 17 in stage 126.0 failed 1 times, most recent failure: Lost task 17.0 in stage 126.0 (TID 5837) (vm-f6e89846 executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container from a bad node: container_1693230454282_0001_01_000002 on host: vm-f6e89846. Exit status: 143. Diagnostics: [2023-08-28 14:48:52.357]Container killed on request. Exit code is 143\n[2023-08-28 14:48:52.365]Container exited with a non-zero exit code 143. \n[2023-08-28 14:48:52.388]Killed by external signal\n.\nDriver stacktrace:\n\tat com.microsoft.dataflow.FileStoreExceptionHandler$.extractRootCause(FileStoreExceptionHandler.scala:36)\n\tat com.microsoft.dataflow.transformers.DefaultFileWriter.$anonfun$write$16(FileStore.scala:1206)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n\tat scala.util.Try$.apply(Try.scala:213)\n\tat com.microsoft.dataflow.transformers.DefaultFileWriter.$anonfun$write$15(FileStore.scal"}
This is the detailed error at the Activity Name level:
{ "errorCode": "BadRequest", "message": "Operation on target Fact Sales Load failed: Operation on target df_factSales failed: {\"StatusCode\":\"DF-Executor-InternalServerError\",\"Message\":\"Job failed due to reason: at Sink 'FinalFactDel': Failed to execute dataflow with internal server error, please retry later. If issue persists, please contact Microsoft support for further assistance\",\"Details\":\"org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 60.0 failed 1 times, most recent failure: Lost task 9.0 in stage 60.0 (TID 2819) (vm-75e51012 executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container from a bad node: container_1693245939507_0001_01_000002 on host: vm-75e51012. Exit status: 143. Diagnostics: [2023-08-28 18:42:28.371]Container killed on request. Exit code is 143\\n[2023-08-28 18:42:28.373]Container exited with a non-zero exit code 143. \\n[2023-08-28 18:42:28.387]Killed by external signal\\n.\\nDriver stacktrace:\\n\\tat com.microsoft.dataflow.FileStoreExceptionHandler$.extractRootCause(FileStoreExceptionHandler.scala:36)\\n\\tat com.microsoft.dataflow.transformers.DefaultFileWriter.$anonfun$write$16(FileStore.scala:1206)\\n\\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\\n\\tat scala.util.Try$.apply(Try.scala:213)\\n\\tat com.microsoft.dataflow.transformers.DefaultFileWriter.$anonfun$write$15(FileStore.scala:11\"}", "failureType": "UserError", "target": "rptSalesDailyLoad", "details": "" } -
QuantumCache 20,366 Reputation points Moderator
2023-08-28T21:01:28.1366667+00:00 Sorry for the inconvenience caused in this matter!
For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Email Body:
Thread URL: <Link to this thread. >
Subscription ID: <>
Email Subject: Attn Satish Boddu
Please let me know once you have done the same.
-
Ryan Wright 0 Reputation points
2023-08-31T14:01:38.87+00:00 Hi @satishBoddu-MSFT I do not believe I have the ability to open a support ticket. We do not have an Azure support subscription. How can we move forward?
-
Ryan Wright 0 Reputation points
2023-08-31T14:09:51.1233333+00:00 Have sent an email per your information on August 28, 2023 at 2:01pm above. Please confirm receipt. Thank you.
-
QuantumCache 20,366 Reputation points Moderator
2023-08-31T15:18:10.01+00:00 @Ryan Wright Thank you!
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 86%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELH%3C/text%3E%3C/svg%3E)
Lars Heinrichs 5 Reputation points2024-01-24T14:12:05.04+00:00 Has this been resolved?
We are facing the same problem since around 2 weeks, without changes to any pipelines. We tried scaling up the cluster, to make sure the execution is not killed due to a resource usage threshold, but ended up with the same result.
Sign in to comment
Sign in to answer