Getting error while executing a pipeline "StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Not started","Details":"java.lang.Throwable: Not started

Ainuddin Khan 1

the overall error is :
Operation on target Generate_WishlistGroupData failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Not started","Details":"java.lang.Throwable: Not started\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply$mcV$sp(FlowRunner.scala:175)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:168)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1$$anonfun$1.apply(FlowRunner.scala:168)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply$mcV$sp(FlowRunner.scala:168)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:160)\n\tat com.microsoft.dataflow.FlowCode$$anonfun$run$1.apply(FlowRunner.scala:160)\n\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)\n\tat scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)\n\tat scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:133"}

We are not able to figure out the reason why it is being failed.

Could you please help me on how to overcome this issue?

Thanks!

Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-08-10T19:56:19.223+00:00

Hello @Ainuddin Khan ,

Thanks for the question and using MS Q&A platform.

Could you please provide us with the below details on the pipeline run?

1) source and destination datasets
2) pipeline activities
3) Integration run times(auto-resolve?)
4) any data flow?
5) Is this failing in every pipeline run? Or intermittent?
6) where exactly did the failure happen?

I am looking forward to hearing from you.
Ainuddin Khan 1 Reputation point

2022-08-12T04:01:56.98+00:00
@Bhargava-MSFT ,

there are multiple source , of type azure sql and azure cosmos

a lot of activities has been used. I am attaching an image.

AutoResolveIntegrationRuntime

yes one dataflow. I am attaching image.

yes, failing everytime

We are not able to identify the exact place of error.
Ainuddin Khan 1 Reputation point

2022-08-12T04:11:20.967+00:00

@Bhargava-MSFT
clean_up SP is working fine.
Once I got a different following error message :
Operation on target Generate_WishlistGroupData failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'StudentSavedWishlistSink': Job aborted due to stage failure: Task 0 in stage 36.0 failed 1 times, most recent failure: Lost task 0.0 in stage 36.0 (TID 36, vm-e3541794, executor 1): java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:109)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.scan_writeFields_2_1$(Unknown Source)\n\tat
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage26.processNext(Unknown Source)\n\tat
Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-08-15T18:20:18.31+00:00

Hi @Ainuddin Khan ,
Thank you for sharing the details. also, could you please share the pipeline run ids for this issue?
Ainuddin Khan 1 Reputation point

2022-08-16T04:52:40.087+00:00
@Bhargava-MSFT
These are the some of run ids:

0de8593c-d159-4dce-9528-0aa617e6dc50

35801ccb-84f5-4159-9071-6a5bdd141620

8868a003-afc9-4a99-b89c-e24cff09999f

6e826c23-59e6-4758-9faf-0f396dfc65f9
Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-08-16T14:58:49.413+00:00

Hello @Ainuddin Khan ,
Thank you for providing the runids. I will work with my internal team and get back to you.
Ainuddin Khan 1 Reputation point

2022-08-17T05:05:51.317+00:00

@Bhargava-MSFT
Thanks,
Please provide the cause of failure.
Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-08-17T21:39:44.737+00:00

Hello @Ainuddin Khan ,
The issue is being hot fixed today(the issue is due to a product bug). Thank you for your patience.
Ainuddin Khan 1 Reputation point

2022-08-22T04:00:44.117+00:00

@Bhargava-MSFT
Is this issue fixed ? Please let me know.
Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-08-22T20:12:13.843+00:00

Hello @Ainuddin Khan ,
Yes, this is fixed.

1 answer

Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-08-16T22:37:32.1+00:00

Hello @Ainuddin Khan ,
The product team identified the problem, and they are working on it. I will keep you posted once the fix is applied.

Latest Update on Aug 17th 2022: This issue is being hot fixed by the product team today. If you ever notice this issue, please feel free to let us know. Thank you for your patience.
Please sign in to rate this answer.
Ainuddin Khan 1 Reputation point

2022-09-04T05:38:11.39+00:00

@Bhargava-MSFT
I am still getting error while executing my pipeline. Please have a look.

RunId : 78bd802e-ad1d-4a34-a447-766e653bb170

The error is:
Operation on target Generate_WishlistGroupData failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'StudentSavedWishlistSink': Job aborted due to stage failure: Task 0 in stage 30.0 failed 1 times, most recent failure: Lost task 0.0 in stage 30.0 (TID 37, vm-7e734586, executor 1): java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:109)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.scan_writeFields_2_1$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage25.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage26.processNext(Unknown Source)\n\tat org.apache.spark.","Details":"org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 30.0 failed 1 times, most recent failure: Lost task 0.0 in stage 30.0 (TID 37, vm-7e734586, executor 1): java.lang.NullPointerException

Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-09-04T18:13:02.167+00:00

Hello @Ainuddin Khan ,
Thanks for the reply.
The Product team fixed the issue that has the error "{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Not started","Details":"java.lang.Throwable: Not started
As per your new error, this needs further troubleshooting.

Could you please try the below and see if it helps?

Re-import projections from the source

check the datatypes at the sink are the same as the destination SQL

if this didn't help, could you please consider opening a new thread for further investigation?

Bhargava-MSFT 30,896 Reputation points Microsoft Employee

2022-09-12T22:17:07.147+00:00

Hello @Ainuddin Khan ,
I am just checking in to see if you have any further questions here.

If this answers your question, please consider accepting the answer by hitting the Accept answer button, as it helps the community.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Getting error while executing a pipeline "StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Not started","Details":"java.lang.Throwable: Not started

1 answer

Your answer