Data Flow failed with : Livy Id=[0] Job failed during run time with state=[dead].

Question

Data Flow failed with : Livy Id=[0] Job failed during run time with state=[dead].

Heta Desai 357

Dataflow that pulls data from Azure sql database got failed with below error. I am executing the Dataflow inside a loop to pull multiple table from azure sql database. for few tables dataflow was running perfectly and after that it got failed.

Livy Id=[0] Job failed during run time with state=[dead].

I have attached a log for dataflow.

Azure-IR-11-application_1678861376088_0001-Driver (stderr) - stderr-2023-03-15-09.log

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-16T21:46:06.0333333+00:00

Hello Heta Desai,

As per your log, an executor on node "vm-c0d53135" was lost due to a container from a bad node. This could be related to the backend cluster issue.

I had seen this error before and was able to resolve it when I retried it again.

Did you get a chance to retry the pipeline again?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-18T02:32:21.3066667+00:00

Hello Heta Desai,

I am checking to see if you got a chance to retry the pipeline again.
Heta Desai 357 Reputation points

2023-03-20T05:36:11.8533333+00:00

@Bhargava-MSFT I tried to re-execute the pipeline multiple times to 2 days. It is giving me same error. Even when I debug dataflow it throws below error:

1 answer

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-16T21:46:06.0333333+00:00

Hello Heta Desai,

As per your log, an executor on node "vm-c0d53135" was lost due to a container from a bad node. This could be related to the backend cluster issue.

I had seen this error before and was able to resolve it when I retried it again.

Did you get a chance to retry the pipeline again?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-18T02:32:21.3066667+00:00

Hello Heta Desai,

I am checking to see if you got a chance to retry the pipeline again.
Heta Desai 357 Reputation points

2023-03-20T05:36:11.8533333+00:00

@Bhargava-MSFT I tried to re-execute the pipeline multiple times to 2 days. It is giving me same error. Even when I debug dataflow it throws below error:

Answer 1

Bhargava-MSFT 31,261 Microsoft Employee Moderator

Hello Heta Desai,

Thanks for the reply and providing the error details.

After checking with my internal team, here is a possible cause and resolution for this issue.

This error occurs when a pipeline is debugged with debug session and dies before the dataflow activity starts. For example, the pipeline has other activities before the dataflow run starts.

In general, the debug session has a TTL of 1 hour.

Can you please perform the below and debug your pipeline?

Please increase the timeout of the debug session to 4 hours to keep the debug session alive for a longer time.

If this still doesn't resolve your issue, please let me know your pipeline run ID and the synapse region for further troubleshooting.

User's image

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-28T00:12:08.1033333+00:00

Hello Heta Desai,

I am checking to see if you got a chance to look into my above response.

Share via

Data Flow failed with : Livy Id=[0] Job failed during run time with state=[dead].

1 answer

Your answer