Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
Discussed on call with MSFT, it's not possible to use spot VMs in HDInsight
How to use spot instances with HDInsight
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 21%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E3%3C/text%3E%3C/svg%3E)
31610895
20
Reputation points
Hi,
- Is there any way to use Spot VMs in HDInsight ?
- How to determine which VMs are spot VMs in this list - https://azure.microsoft.com/en-in/pricing/details/virtual-machines/linux/ ?
- Is there any max limit on number of Spot VMs that can be used in HDInsight cluster ?
Thanks,
Akshit Mehta
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 11%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETH%3C/text%3E%3C/svg%3E)
Tech-Hyd-1989 5,796 Reputation points2023-08-16T10:36:41.22+00:00 Hello 31610895
Yes, it is possible to use Spot VMs in HDInsight. Spot VMs are Azure virtual machines that are available at a discounted price. They are ideal for workloads that can handle interruptions, such as batch processing jobs or data analytics.
To use Spot VMs in HDInsight, you need to create a custom cluster. When you create a custom cluster, you can specify that you want to use Spot VMs. You can also specify the maximum price that you are willing to pay for Spot VMs.
Here are the steps on how to use Spot VMs in HDInsight:
- Go to the Azure portal and sign in with your Azure account.
- Click the HDInsight blade.
- In the left navigation pane, click Clusters.
- Click the Create cluster button.
- In the Cluster configuration section, select the Custom cluster type.
- In the VM size section, select the Spot VM option.
- In the Maximum price section, enter the maximum price that you are willing to pay for Spot VMs.
- Click the Create button.
Once the cluster is created, it will start using Spot VMs. If the price of Spot VMs goes above your maximum price, the cluster will be evicted.
Here are some things to keep in mind when using Spot VMs in HDInsight:
- Spot VMs are not guaranteed to be available. If there is no Spot VM capacity available, your cluster will be created using on-demand VMs.
- Spot VMs can be evicted at any time. If the price of Spot VMs goes above your maximum price, your cluster will be evicted.
- Spot VMs are not eligible for Azure Reserved Instances discounts.
Overall, Spot VMs can be a great way to save money on your HDInsight cluster. However, it is important to understand the limitations of Spot VMs before you use them.
Yes, there is a maximum limit on the number of Spot VMs that can be used in an HDInsight cluster. The maximum number of Spot VMs that can be used in a cluster depends on the cluster type and the region where the cluster is deployed.
The following table shows the maximum number of Spot VMs that can be used in an HDInsight cluster:
Hadoop All regions 32 Spark All regions 16 Hive All regions 16 Storm All regions 8 HBase All regions 8 Kafka All regions 8 drive_spreadsheetExport to Sheets
It is important to note that these are the maximum limits. The actual number of Spot VMs that can be used in a cluster may be less than the maximum limit, depending on the availability of Spot VM capacity.
If you need to use more than the maximum number of Spot VMs in a cluster, you can create multiple clusters. However, this can be expensive, as you will be paying for the resources used by all of the clusters, even if some of the clusters are not fully utilized.
A better option is to use reserved instances. Reserved instances are a type of Azure VM that can be purchased at a discounted price. They are available for a one-year or three-year term. Reserved instances are not eligible for Spot pricing, but they are guaranteed to be available.
If you need to use a large number of VMs for your HDInsight cluster, it is a good idea to use a combination of Spot VMs and reserved instances. This will allow you to save money on your VM costs while still ensuring that you have the resources you need.
-
31610895 20 Reputation points
2023-08-16T16:52:06.2933333+00:00 @Subrotho Das thanks for answering the query in detail. I still have a few questions -
- How can i identify from list of VMs that a particular VM is spot instance (Point 2 in posted question)
- I tried following steps to use spot VMs, but i didn't find
Customcluster type option while creating cluster. The options that i see for cluster type (while creating new cluster) are -Hadoop,Hbase,Spark,Kafka,Interactive Query
-
31610895 20 Reputation points
2023-08-17T03:20:30.9833333+00:00 @Subrotho Das thanks for answering the query in detail. I still have a few questions -
- How can i identify from list of VMs that a particular VM is spot instance (Point 2 in posted question)
- I tried following steps to use spot VMs, but i didn't find
Customcluster type option while creating cluster. The options that i see for cluster type (while creating new cluster) are -Hadoop,Hbase,Spark,Kafka,Interactive Query
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 45%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESZ%3C/text%3E%3C/svg%3E)
shlim@zenithn.com 6 Reputation points2023-08-17T06:34:57.5766667+00:00 I think he just got the answer from chatGPT.
I don't think it's gonna be working.
Sign in to comment
2 answers
Sort by: Most helpful
-
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT 31,116 Reputation points Microsoft Employee2023-08-22T19:02:32.2366667+00:00 Hello 31610895,
I'm glad that you could find the answer you are looking for. Thank you for informing us that it is not possible to use spot VMS in HDinsights.
Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.
How to use Spot VMs in HDInsight:
Answer:
Using Azure Spot Virtual Machines allows you to take advantage of our unused capacity at a significant cost savings. At any point in time when Azure needs the capacity back, the Azure infrastructure will evict Azure Spot Virtual Machines.
SPOT VMS are only applicable to:
✔️ Linux VMs ✔️ Windows VMs ✔️ Flexible scale sets ✔️ Uniform scale setsFor ex:
To enable SPOT instances on the virtual machine scale set, we need to enable the option "Run with Azure spot discount."
Whereas on the HDInsights cluster VMs, there is no option to enable "spot instances."
We can conclude that SPOT instances are not supported on the HDInsights cluster VMS.
VM scale set:
HDInsight cluster VMs:
I hope this gives a clear understanding of the SPOT instances of HDInsights cluster VMs.
Reference document:
https://learn.microsoft.com/en-us/azure/virtual-machines/spot-vms
Please remember to "Accept Answer" if any answer/reply helped so that others in the community facing similar issues can easily find the solution.