Pipeline failed at dataflow with below error

Question

Operation on target dataflow_failed: Livy job state dead caused by unknown error, details: '[plugins.xxxxxxx.IRTest.666 WorkspaceType:

Answer 1

Hello @Anonymous ,

Welcome to the MS Q&A platform.

"Livy job state dead caused by unknown error" is a transient error. The dataflow uses a spark cluster at the backend and this error is thrown from the spark cluster.
There could be a network issue or resource church issue at the backend cluster or an outage issue.

Below are some recommendations to avoid this issue.
 
Configure the retry count in the dataflow setting to 3 as per your data size.
Configure the retry timeout to 30 sec.

If you are seeing the issue quite often, please log a ticket with the support team and provide the pipeline runid for further troubleshooting.

I hope this helps. Please let me know if you have any further questions.

------------------------------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

@Bhargava-MSFT

I tried changed retry count in the dataflow setting to 3 and timeout to 30 sec.

I am still having the same issues.

See my thread:

[https://learn.microsoft.com/en-us/answers/questions/1164358/adf-dataflow-stuck-in-progress?page=1&orderby=helpful&comment=answer-1166946#newest-answer-comment