databricks cluster schedule start time

Question

hello,

Is there a way to schedule the cluster to start and stop at specific time?
It looks like there is only the option to set it for how long it is to be running for

thanks

Answer 1

@arkiboys - Thanks for the question and using MS Q&A platform.

Yes, it's possible to start and stop the cluster at specific time using Azure Automation.

Example: Start the databricks cluster as per your team works from 8AM to 6PM on weekdays using Azure Automation.

To start at 8 AM you can use PowerShell runbook in Azure Automation to start your cluster as per the scheduled time as shown below:

PowerShell runbook should be as shown below:

$accessToken = "<Personal_Access_Token>"
$apiUrl = "<Azure_Databricks_Endpoint_URL>"
Set-DatabricksEnvironment -AccessToken $accessToken -ApiRootUrl $apiUrl
Start-DatabricksCluster -ClusterID "<Cluster_ID>"

To stop at 6 PM you can set the property Terminate after 600 minutes of inactivity.

Note: If your Business Hours (8AM to 6PM which means 10 Hours x 60 minutes) you can set the property Terminate after 600 minutes of inactivity as shown below:

This Tutorial: Start Azure Databricks clusters during business hours walks you through the creation of a PowerShell Workflow runbook to start Azure Databricks clusters during business hours in Azure Automation.

For more details, refer https://stackoverflow.com/questions/69292078/start-azure-databricks-clusters-during-business-hours.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.