Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Pipeline failed
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 89%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rajat Katiyar (INFOSYS LIMITED)
21
Reputation points
The pipeline was running successfully till yesterday but today it got failed suddenly. the source is of delta file and the sink is the parquet file. it shows the error like job failed due to the reason at Sink 'ToBLOBAsParquet'.
Azure Data Factory
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator2021-10-15T07:32:25.59+00:00 Hi @Rajat Katiyar (INFOSYS LIMITED) ,
Welcome to Microsoft Q&A forum and thanks for reaching out here.
To better assist on this issue, could you please share the complete error message along with a screen shot of it? Also please do share the failed pipeline and activity runID for further analysis.
But as per your statement that it only failed today and looking at the error message I suspect it could be related to some data issue. I would recommend you to please check if there is any mismatch in data types or column mapping in sink transformation as the error clearly says
Sink 'ToBLOBAsParquet'Looking at the data and the sink mapping should give you some pointers about the root cause.
Do let us know how it goes. If you still block please do share above requested details.
Thank you
-
Rajat Katiyar (INFOSYS LIMITED) 21 Reputation points
2021-10-16T05:33:30.427+00:00 {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'ToBLOBAsParquet': org.apache.spark.SparkException: Job aborted.","Details":"org.apache.spark.SparkException: Job aborted.\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)\n\tat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:152)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:148)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:176)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:1"}
this is the error msg i received.
-
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator
2021-10-21T19:00:03.687+00:00 Hi @Rajat Katiyar (INFOSYS LIMITED) ,
Thanks for the details. Could you please share the failed pipeline and activity runID for deeper analysis.
Thanks
-
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator
2021-10-25T18:13:28.913+00:00 Hi there,
We still have not heard back from you. Just wanted to check if you are still facing the issue or need assistance on this? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.
-
Rajat Katiyar (INFOSYS LIMITED) 21 Reputation points
2021-10-26T03:49:12.587+00:00 Hi Kranthi, want to sync with you can you please sync with me on teams?
-
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator
2021-10-29T05:18:02.643+00:00 Hi @Rajat Katiyar (INFOSYS LIMITED) ,
Could you please share the support ticket number here so that we can track it internally.
Thank you
Sign in to comment
Sign in to answer