This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 48%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
the overall error is :
Operation on target Generate_WishlistGroupData failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Not started","Details":"java.lang.Throwable: Not started\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply$mcV$sp(FlowRunner.scala:175)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:168)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:168)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply$mcV$sp(FlowRunner.scala:168)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:160)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:160)\n\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)\n\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)\n\tat scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:133"}
We are not able to figure out the reason why it is being failed.
Could you please help me on how to overcome this issue?
Thanks!
An Azure service for ingesting, preparing, and transforming data at scale.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Hello @Ainuddin Khan ,
Thanks for the question and using MS Q&A platform.
Could you please provide us with the below details on the pipeline run?
1) source and destination datasets
2) pipeline activities
3) Integration run times(auto-resolve?)
4) any data flow?
5) Is this failing in every pipeline run? Or intermittent?
6) where exactly did the failure happen?
I am looking forward to hearing from you.
@Bhargava-MSFT
clean_up SP is working fine.
Once I got a different following error message :
Operation on target Generate_WishlistGroupData failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'StudentSavedWishlistSink': Job aborted due to stage failure: Task 0 in stage 36.0 failed 1 times, most recent failure: Lost task 0.0 in stage 36.0 (TID 36, vm-e3541794, executor 1): java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:109)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.scan_writeFields_2_1$(Unknown Source)\n\tat
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage26.processNext(Unknown Source)\n\tat
Hi @Ainuddin Khan ,
Thank you for sharing the details. also, could you please share the pipeline run ids for this issue?
@Bhargava-MSFT
These are the some of run ids:
Hello @Ainuddin Khan ,
Thank you for providing the runids. I will work with my internal team and get back to you.
@Bhargava-MSFT
Thanks,
Please provide the cause of failure.
Hello @Ainuddin Khan ,
The issue is being hot fixed today(the issue is due to a product bug). Thank you for your patience.
@Bhargava-MSFT
Is this issue fixed ? Please let me know.
Hello @Ainuddin Khan ,
Yes, this is fixed.
Hello @Ainuddin Khan ,
The product team identified the problem, and they are working on it. I will keep you posted once the fix is applied.
Latest Update on Aug 17th 2022: This issue is being hot fixed by the product team today. If you ever notice this issue, please feel free to let us know. Thank you for your patience.
@Bhargava-MSFT
I am still getting error while executing my pipeline. Please have a look.
RunId : 78bd802e-ad1d-4a34-a447-766e653bb170
The error is:
Operation on target Generate_WishlistGroupData failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'StudentSavedWishlistSink': Job aborted due to stage failure: Task 0 in stage 30.0 failed 1 times, most recent failure: Lost task 0.0 in stage 30.0 (TID 37, vm-7e734586, executor 1): java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:109)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.scan_writeFields_2_1$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage26.processNext(Unknown Source)\n\tat org.apache.spark.","Details":"org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 30.0 failed 1 times, most recent failure: Lost task 0.0 in stage 30.0 (TID 37, vm-7e734586, executor 1): java.lang.NullPointerException
Hello @Ainuddin Khan ,
Thanks for the reply.
The Product team fixed the issue that has the error "{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Not started","Details":"java.lang.Throwable: Not started
As per your new error, this needs further troubleshooting.
Could you please try the below and see if it helps?
if this didn't help, could you please consider opening a new thread for further investigation?
Hello @Ainuddin Khan ,
I am just checking in to see if you have any further questions here.
If this answers your question, please consider accepting the answer by hitting the Accept answer button, as it helps the community.