Azure Data Factory Dataflow job failed due to reason: not started

Question

Hello everyone, I have an ADF job which will be triggered hourly with a concurrency of 9. However, I always have a few failed jobs out of 9 and they came with the following error: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Not started","Details":"java.lang.Throwable: Not started\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1$$anonfun$2.apply$mcV$sp(FlowRunner.scala:201)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1$$anonfun$2.apply(FlowRunner.scala:194)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1$$anonfun$2.apply(FlowRunner.scala:194)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply$mcV$sp(FlowRunner.scala:194)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:185)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:185)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply$mcV$sp(FlowRunner.scala:185)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:177)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:177)\n\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Futur"} Upon some reruns it will eventually be successful. However, this is a very irritating error message as I cannot know what is exactly going on. A failed job is successful after some reruns, so I guess the job itself is error free. Any help will be appreciated, thank you in advanced. Best regards, Patrick

Answer 1

Hello Patrick Wong, Welcome to the MS Q&A platform. As you mentioned, the job succeeded after multiple runs; it seems like this is a transient issue. Can you please tell me when did you notice the issue for the first time? Is this specific to a single pipeline? A similar thread has been discussed on the below URL. Please follow the steps mentioned in the answer section and see if it helps. https://learn.microsoft.com/en-us/answers/questions/959963/azure-synapse-error-job-failed-due-to-reason-not-s