How to switch Databricks Compute resources OS Disk to SSD

Question

Hi,

our databricks compute resource are running OS Disk HDD, therefor the start up take too long.

How can we switch it to ssd.

Answer 1

Hi Fabian Börner,

In Azure Databricks, the OS disk for driver and worker nodes is managed entirely by the Databricks service. Because the compute nodes are provisioned and recreated automatically by the service, there is no supported way to directly change the OS disk type (for example from Standard HDD to Standard SSD or Premium SSD) for an existing Databricks cluster. Any manual changes at the underlying VM or disk level are not supported and would be lost when the cluster is restarted, scaled, or recreated.

Cluster startup time is usually influenced by factors other than the OS disk type. For example, delays can occur due to Azure capacity allocation in the region, large cluster sizes, cold starts when no pre-provisioned nodes are available, retries when using spot instances, or initialization scripts that install packages during cluster startup.

To improve startup performance, the recommended approach is to focus on cluster configuration rather than OS disk settings. Using newer VM families for your node type, enabling instance pools so nodes are pre-allocated and ready to attach to clusters, and minimizing work done in initialization scripts can significantly reduce startup time. Instance pools are particularly helpful because they keep idle nodes available, which avoids the delay of provisioning new VMs each time a cluster starts.

If you’d like, please share the node type currently used by your cluster, whether instance pools are configured, and whether you are using initialization scripts. With that information, we can help review your configuration and suggest optimizations to improve startup time.

Hope this helps, Please let us know if you have any questions and concerns.