Azure Synapse Mapping Dataflow - Job failed due to reason: Not started.

Question

Azure Synapse Mapping Dataflow - Job failed due to reason: Not started.

Christopher Geckert 91

Hello,

since yesterday we've got an issue with two mapping Data Flows in Azure Synapse:

{"message":"Job failed due to reason: Not started. Details:java.lang.Throwable: Not started\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply$mcV$sp(FlowRunner.scala:175)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:168)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:168)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply$mcV$sp(FlowRunner.scala:168)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:160)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:160)\n\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)\n\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)\n\tat scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:133","failureType":"UserError","target":"Data flow1","errorCode":"DFExecutorUserError"}

Is there any issue with the debug clusters or something like that? Also from time to time the Pipes and Dataflow just don't start and Queue forever.

Bhargava-MSFT 31,246 Reputation points Microsoft Employee Moderator

2022-08-15T18:22:19.313+00:00

Hello @Christopher Geckert ,

Thanks for the question and using MS Q&A platform.

Could you please share the pipeline run ids for this issue?
Bhargava-MSFT 31,246 Reputation points Microsoft Employee Moderator

2022-08-16T22:48:36.06+00:00

Hello @Christopher Geckert ,
The product team identified the problem, and they are working on it. I will keep you posted once the fix is applied.
Thank you for being so patient.
Christopher Geckert 91 Reputation points

2022-08-17T08:07:54.84+00:00

Hello,
@Bhargava-MSFT did you already change something? We now encounter:
{"message":"Job failed due to reason: None.get. Details:java.util.NoSuchElementException: None.get\n\tat scala.None$.get(Option.scala:347)\n\tat scala.None$.get(Option.scala:345)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:517)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:515)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat com.microsoft.dataflow.FlowCode.com$microsoft$dataflow$FlowCode$$recurseLineageNode(FlowRunner.scala:515)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:518)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:515)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(A","failureType":"UserError","target":"Data flow1","errorCode":"DFExecutorUserError"}

Pipeline Run ID: fbba4dcf-ce6b-4ffe-ad30-ab745bb311d8
Dataflow RunID: 993b49e2-6fa5-4e35-94ba-273714bcb527
Sandra Aerts 6 Reputation points

2022-09-15T11:18:34.47+00:00

Is this the same issue? Today in my Data Factory:

failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: None.get","Details":"java.util.NoSuchElementException: None.get\n\tat scala.None$.get(Option.scala:347)\n\tat scala.None$.get(Option.scala:345)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:547)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:545)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat com.microsoft.dataflow.FlowCode.com$microsoft$dataflow$FlowCode$$recurseLineageNode(FlowRunner.scala:545)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:548)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:545)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(A"}

The underlying data flow succeeded. The pipeline holds nothing else, just this 1 dataflow. Debug mode.
Bhargava-MSFT 31,246 Reputation points Microsoft Employee Moderator

2022-09-15T21:19:20.563+00:00

Hello @Sandra Aerts ,
Sorry, This is not the same error. The product team fixed the issue of the error message: ""message":"Job failed due to reason: Not started"

1 answer

Your answer

Bhargava-MSFT 31,246 Reputation points Microsoft Employee Moderator

2022-08-15T18:22:19.313+00:00

Hello @Christopher Geckert ,

Thanks for the question and using MS Q&A platform.

Could you please share the pipeline run ids for this issue?
Bhargava-MSFT 31,246 Reputation points Microsoft Employee Moderator

2022-08-16T22:48:36.06+00:00

Hello @Christopher Geckert ,
The product team identified the problem, and they are working on it. I will keep you posted once the fix is applied.
Thank you for being so patient.
Christopher Geckert 91 Reputation points

2022-08-17T08:07:54.84+00:00

Hello,
@Bhargava-MSFT did you already change something? We now encounter:
{"message":"Job failed due to reason: None.get. Details:java.util.NoSuchElementException: None.get\n\tat scala.None$.get(Option.scala:347)\n\tat scala.None$.get(Option.scala:345)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:517)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:515)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat com.microsoft.dataflow.FlowCode.com$microsoft$dataflow$FlowCode$$recurseLineageNode(FlowRunner.scala:515)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:518)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:515)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(A","failureType":"UserError","target":"Data flow1","errorCode":"DFExecutorUserError"}

Pipeline Run ID: fbba4dcf-ce6b-4ffe-ad30-ab745bb311d8
Dataflow RunID: 993b49e2-6fa5-4e35-94ba-273714bcb527
Sandra Aerts 6 Reputation points

2022-09-15T11:18:34.47+00:00

Is this the same issue? Today in my Data Factory:

failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: None.get","Details":"java.util.NoSuchElementException: None.get\n\tat scala.None$.get(Option.scala:347)\n\tat scala.None$.get(Option.scala:345)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:547)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:545)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat com.microsoft.dataflow.FlowCode.com$microsoft$dataflow$FlowCode$$recurseLineageNode(FlowRunner.scala:545)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:548)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$com$microsoft$dataflow$FlowCode$$recurseLineageNode$1.apply(FlowRunner.scala:545)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(A"}

The underlying data flow succeeded. The pipeline holds nothing else, just this 1 dataflow. Debug mode.
Bhargava-MSFT 31,246 Reputation points Microsoft Employee Moderator

2022-09-15T21:19:20.563+00:00

Hello @Sandra Aerts ,
Sorry, This is not the same error. The product team fixed the issue of the error message: ""message":"Job failed due to reason: Not started"

Answer 1

Bhargava-MSFT 31,246 Microsoft Employee Moderator

Hello @Christopher Geckert ,

The product team identified the problem and they are working on it. I will keep you posted once the fix is applied.

Latest Update on Aug 17th, 2022: This issue is being hot fixed by the product team today. If you ever notice this issue, please feel free to let us know. Thank you for your patience.

Bhargava-MSFT 31,246 Reputation points Microsoft Employee Moderator

2022-08-20T03:06:22.39+00:00

Hello @Christopher Geckert ,

I just wanted to check if the issue is resolved and have any further questions. Please let me know.
Alessandro Ribul Moro 87 Reputation points

2022-10-14T09:45:05.907+00:00

Details: Job failed due to reason: None.get

Dataflow is starting.
Data is written to sink correctly.
Dataflow throws error.
Alessandro Ribul Moro 87 Reputation points

2022-10-14T12:10:16.367+00:00

Okay after hours of trying out what could be the problem, it seems like the "new" cast block is not working with an assert together, leading to the problem that sink to error path do not work correctly.
After switching the cast to derivedColumn, the not successful transformed data rows are written successfully to error path.
Seems like the cast operation is the problem.
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-cast
Milos Przedpelski 0 Reputation points

2024-04-16T18:34:33.7766667+00:00

Has this issue been resolved? I am getting the very same error message!

Job failed due to reason: com.microsoft.dataflow.broker.InvalidParameterTypeException: Invalid type for parameter: srcFileId

Share via

Azure Synapse Mapping Dataflow - Job failed due to reason: Not started.

1 answer

Your answer