Right configuration for 5000 concurrent query request on ADX Cluster - Size of Cluster and Optimal number of nodes and Configuration of policy

Question

Now I'm seeking assistance to finalize our approach for efficiently managing 5000 concurrent queries on an ADX cluster through the Kusto client library. I require guidance on selecting the right cluster, configuring nodes, adjusting settings, and evaluating associated costs. Your expertise will greatly aid in determining the optimal cluster size, node setup, and settings to ensure smooth query performance while keeping expenses in check. Your support in conducting a comprehensive cost analysis will be invaluable in making informed decisions, allowing us to successfully handle the query load on the ADX cluster.

Answer 1

Hi Deena Nath | MAQ Software,

Thank you for posting query in Microsoft Q&A Platform.

To efficiently manage 5000 concurrent queries on an ADX cluster through the Kusto client library, we need to consider several factors such as cluster size, node setup, and settings.

First, we need to determine the right cluster size based on the expected query load. We can use the Azure Data Explorer Cost Estimator tool to estimate the cost of running the cluster. The tool will suggest a VM size and number of instances for your cluster based on the workload type and the amount of data queried actively.

Second, we need to configure the nodes in the cluster to optimize query performance**.** We can consider using a combination of read and write nodes to balance the query load and ensure smooth performance. We can also adjust the number of nodes based on the expected query load and the size of the data being queried.

Third, we need to adjust the cluster settings to optimize query performance. We can consider enabling caching to reduce query latency and adjusting the query timeout settings to ensure that long-running queries do not impact the performance of other queries.

Finally, we need to conduct a comprehensive cost analysis to evaluate the associated costs of running the cluster. We can use the Azure Data Explorer Cost Estimator tool to estimate the cost of running the cluster based on the expected query load and the size of the data being queried. We can also consider using Azure Advisor to optimize the cost of running the cluster.

I hope this information helps you in determining the optimal cluster size, node setup, and settings to ensure smooth query performance while keeping expenses in check. If you have any further questions or require additional assistance, please let me know.