Share via

Databricks cluster failure : OutOfMemoryError

NIKHIL KUMAR 101 Reputation points
2023-07-11T17:03:22.0533333+00:00

Getting a continous failures after a couple of days of stream runs with below error message, what could be potential permanent solution for the same ?

Driver restarted - possibly due to an OutOfMemoryError - and this stream has been stopped. Please check the Driver Logs for more details.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,307 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Bhargava-MSFT 31,191 Reputation points Microsoft Employee
    2023-07-11T21:05:43.4566667+00:00

    HelloNIKHIL KUMAR

    "Driver restarted - possibly due to an OutOfMemoryError" indicates the driver node in your Databricks cluster has run out of memory.

    This can be due to insufficient memory allocation on the driver node or inefficient code.

    Did you see any exception messages in the driver logs?

    Could you please try the below?

    Can you edit the Databricks cluster configuration and increase the driver node memory allocation?

    And if you use a smaller cluster, please try with a different size(large cluster) and see if it helps.

    also, see if you can optimize your code

    User's image

    I hope this helps.

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.