cluster performance in ideal time

Question

Hi Expert,

how to avoid the ideal time/sleeping time scenarios on databricks cluster so that it would run smoothly even though there is no traffic. and we should not start all the time and should run continuously

Answer 1

Hello Shambhu Rai,

Databricks provides a setting called "Terminate after X minutes of activity" that can be used to avoid the ideal time/sleeping time scenarios on Databricks cluster.

When enabled, the cluster will terminate after the specified time interval of inactivity (i.e., no running commands or active job runs). This feature is best supported in the latest Spark versions. Learn more

This setting terminates the cluster after the specified number of minutes of inactivity. You can set this value to an appropriate value to avoid the cluster going into sleeping mode.

If you set a high value, the cluster will stay idle even though there is no workload running, but it could result in high costs. so you must set the appropriate value based on your needs

I hope this helps. Please let me know if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.