Data Management Instance Scaling in Azure Data Explorer and VM Reservation Implications

Question

Hello community,

I have a question regarding the scaling of Data Management instances in Azure Data Explorer (ADX), focusing on VM types and cost implications.

Recently, our ADX setup unexpectedly utilized D2 v2 VMs for data management instead of the anticipated D1 v2 VMs, resulting in higher costs. Similarly, I've noticed variations in VM usage (D1, D2, D3) within our cost management reports over different months.

My query is twofold:

1. What factors influence the scaling of Data Management instances within the same engine SKU in ADX? We intended to use D1 v2 consistently but have observed changes to D2 v2 and D3 v2.
2. Given this variability, how can we accurately predict and manage our budget? Is it possible to purchase VM reservations for specific types (e.g., D1 v2) if instances may change dynamically?

Any insights or best practices on these topics would be greatly appreciated.

Thank you!

Best regards,
Wes Verhagen

Answer 1

Hello Wes Verhagen,

Welcome to the Microsoft Q&A forum.

It seems like you have configured the ADX cluster with "Optimized autoscale" https://learn.microsoft.com/en-us/azure/data-explorer/manage-cluster-horizontal-scaling?WT.mc_id=Portal-Microsoft_Azure_Kusto#optimized-autoscale-recommended-option

Optimized autoscale:

Optimized Autoscale is a built-in feature that helps clusters perform their best when demand changes. You can choose to scale your cluster manually to a specific instance count, or via a custom Optimized Autoscale policy that scales based on metric(s) thresholds. Optimized Autoscale enables your cluster to be performant and cost effective by adding and removing instances based on demand

• If the cluster is underutilized, it is scaled in to lower cost without affecting the required performance.
• If the cluster is overutilized, it is scaled out to maintain optimal performance

In your case, the changes in VM types (from D1 v2 to D2 v2 and D3 v2) are likely due to the Optimized Autoscale feature responding to varying demands on your cluster.

To have more control over the scaling and potentially reduce costs, you might want to look into the custom autoscale feature. Custom autoscale allows you to set specific policies and thresholds for scaling based on your needs.

https://learn.microsoft.com/en-us/azure/data-explorer/manage-cluster-horizontal-scaling?WT.mc_id=Portal-Microsoft_Azure_Kusto#custom-autoscale

I hope this helps. Please let me know if you have any further questions.