Is it good decision to use spot instances for databricks worker nodes

Question

Hi Team,

using spot instances for data bricks worker node for cost optimization is good decision or not ..

As far as i know spot instances are immediately evicted when capacity is requested by higher paying customers ..

So using this for production workloads is wise decision or not .. what microsoft recommends for it ..

Thanks,
Mahi

Answer 1

Hello @Rohit Boddu ,

Thanks for the question and using MS Q&A platform.

Azure Spot Virtual Machines are great for workloads that can handle interruptions like batch processing jobs, dev/test environments, large compute workloads, and more.

At any point in time when Azure needs the capacity, the Azure infrastructure will evict Azure Spot VMs. Azure’s eviction policy makes Spot VMs well suited for Azure Databricks, whose clusters are resilient to interruptions for a variety of data and AI use cases, such as ingestion, ETL, stream processing, AI models, batch scoring and more.

Spot VMs pricing and availability varies based on size, region, time of day, and more. When deploying Spot VMs, Azure will allocate the VMs if there is spare capacity available, but there is no SLA. When Azure needs the capacity back, Azure Spot VMs will be evicted with 30-seconds notice.

For more details, refer to Leverage Unused Compute Capacity for Data + AI With Azure Spot Instances and Azure Databricks and Use Azure Spot Virtual Machines.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.