I have a scheduled pipeline in Azure Data Factory which uses Databricks notebooks: For some of the notebooks, I am getting the below error, at the last step of the notebook, which is the step where data is saved on our data lake: This error only occurs when the notebooks are triggered by the pipeline from ADF. When I trigger the notebooks manually, everything works. What is the problem and the solution? Regards,

Issue : When databrick notebooks are triggered by the ADF pipeline get an error "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached" Solution: "The spark driver has stopped unexpectedly" is due to insufficient driver memory. We observed when the error occurs from the ADF pipelines, the notebooks are running concurrently. The error was resolved after changing the Notebooks run to sequential via the ADF pipeline.

Problem solved by: not running notebooks concurrently. My pipeline included running notebooks concurrently.

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

Radu Sitaru (ACCENTURE) 20 Microsoft Employee

I have a scheduled pipeline in Azure Data Factory which uses Databricks notebooks:
User's image

For some of the notebooks, I am getting the below error, at the last step of the notebook, which is the step where data is saved on our data lake:
User's image

This error only occurs when the notebooks are triggered by the pipeline from ADF.

When I trigger the notebooks manually, everything works.

What is the problem and the solution?

Regards,

Bhargava-MSFT 31,016 Reputation points Microsoft Employee

2023-09-12T19:56:03.0433333+00:00
Hello Radu Sitaru (ACCENTURE),

Welcome to the Microsoft Q&A forum.

You will see this error: "The spark driver has stopped unexpectedly" due to insufficient driver memory.

Please check the spark driver memory configuration and review the code to see if there are any operations such as collect(), display(), show() or take() on the data frame causing this issue.

If you have the Spark driver log file, it'd be useful to check what happened and confirm if there was memory pressure.

Resolution:

Increasing the memory config for the driver node should resolve the issue in most cases.

Also, see the driver version and if you can upgrade to the latest version.

https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#driver-node

If there are other cost-related concerns, reviewing the code for the above-mentioned operations and tuning the application can be done.

Recommendations to avoid this issue in the future.

Improve efficiency in the Spark program’s memory management by persisting the Data frame appropriately and explicitly freeing up the Data frame in the cache

Avoid or control the creation of large objects in program code.

Use more powerful VMs to allocate more memory for your job executions

Use a higher driver node instance if you notice GC issues in the driver from your cluster event logs

Detach idle notebooks to free up the heap space after 1 hour

Use native spark dataFrame and avoid toPandas(). This will reduce memory pressure on the Driver Node. And try to use the spark to create the dataframes by using pyspark.pands instead of pandas

Please follow the below Spark memory and Garbage Collection tuning recommendations

https://spark.apache.org/docs/latest/tuning.html#tuning-spark

I hope this helps. Please let me know if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
Bhargava-MSFT 31,016 Reputation points Microsoft Employee

2023-09-13T15:58:50.8366667+00:00

Hello Radu Sitaru (ACCENTURE),

I am checking to see if you got a chance to look into the earlier answer.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Accepted answer

Bhargava-MSFT 31,016 Reputation points Microsoft Employee

2023-09-14T16:36:05.61+00:00

Issue: When databrick notebooks are triggered by the ADF pipeline get an error "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached"

Solution:

"The spark driver has stopped unexpectedly" is due to insufficient driver memory.

We observed when the error occurs from the ADF pipelines, the notebooks are running concurrently.

The error was resolved after changing the Notebooks run to sequential via the ADF pipeline.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Radu Sitaru (ACCENTURE) 20 Reputation points Microsoft Employee

2023-09-14T11:41:01.7+00:00

Problem solved by: not running notebooks concurrently. My pipeline included running notebooks concurrently.
Please sign in to rate this answer.
Bhargava-MSFT 31,016 Reputation points Microsoft Employee

2023-09-14T16:36:01.3566667+00:00

Hello Radu Sitaru (ACCENTURE),

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution if you'd like to "Accept " the answer.

Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Sweety 0 Reputation points

2023-10-17T04:27:31.6266667+00:00

Could you please elaborate this. i am also facing the same issue could you please kindly explain what to do to avoid this error

Sweety 0 Reputation points

2023-10-17T09:24:01.3533333+00:00

Hi could you please share a detailed answer for this like how you overcome this problem
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

1 additional answer

Your answer