How much time it will take for Databricks cluster node vm to shutdown in Azure

Question

Consider the scenario where, ADF pipeline is calling a notebook activity which create a Job cluster in the linked azure databricks. when the job cluster is created and running the workload, we could see respective cluster VMs in the azure portal with random alpha numeric name like 1ff57f094abe4481b353a226993f2801 . Once the cluster has been shut down, how much time will the vms in azure portal will take to shut down and get deleted ?

Answer 1

Hello veerabhadra reddy kovvuri,

Thanks for the question.

I have discussed this with my internal team and per their input.

There is a delay of approximately 7 to 10 minutes between the time you turn off a cluster and the time that Azure starts to delete the underlying VMs. It also depends on the size of the VM, number of VMs and the workload running on the cluster.

The reason behind this is, Databricks does this in case you immediately start a similar cluster, and the service can reuse the existing VMs

You can monitor this from AzureActivity table in the Azure log analytics workspace(monitoring).

I hope this helps. Please let me know if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.