How can I clear the cache in my Databricks Cluster

Ive tried many articles and links without success.

Has anyone managed to successfully clear the cache on their Databricks cluster? This causes many issues as my memory is near its peak, meaning my notebooks often crashes. Any help or advised would be much appreciated.

User's image

Bhargava-MSFT 30,991 Reputation points Microsoft Employee

2024-06-18T17:01:11.2466667+00:00

Hello richardthomas66@hotmail.com,

Have you tried spark.catalog.clearCache() method? This will clear the cached data in Spark. This method removes all cached data from memory and disk.

You can also see the below threads:

https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/td-p/24340

https://community.databricks.com/t5/data-engineering/unable-to-clear-cache-using-a-pyspark-session/td-p/8568

https://learn.microsoft.com/en-us/answers/questions/1664091/how-to-reduce-unnecessary-high-memory-usage-in-a-d

I hope this helps.
Bhargava-MSFT 30,991 Reputation points Microsoft Employee

2024-06-19T19:35:52.0033333+00:00

Hello richardthomas66@hotmail.com,

I am checking to see if you have any further questions here.

Share via

How can I clear the cache in my Databricks Cluster

Your answer