Why can't set spark.cleaner.referenceTracking.cleanCheckpoints in Databricks

Question

I am using datafame.checkpoint(). I'd like to set spark.cleaner.referenceTracking.cleanCheckpoints to true. When I use spark.conf.set('spark.cleaner.referenceTracking.cleanCheckpoints', 'true'), I got "Cannot modify the value of a Spark config: spark.cleaner.referenceTracking.cleanCheckpoints". Why is it so? What's the default value for spark.cleaner.referenceTracking.cleanCheckpoints in Databricks?

Answer 1

Hello @Ping Xiao ,

Welcome to the Microsoft Q&A platform.

Note: Some of the spark properties that should be setup on cluster start.

By default, spark.cleaner.referenceTracking.cleanCheckpoints is set to false.

If you want to set spark.cleaner.referenceTracking.cleanCheckpoints is set to true. you should set it on the Spark Config under Advanced Options in the cluster configuration.

Before and after configuring the Spark configuration under Advanced options in the cluster:

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.