remainingCores is too small error. Spark and Kafka cluster on HDInsight problem

Question

I tried to create a Kafka and Spark cluster in the HDInsight using either of templates in these links:
https://learn.microsoft.com/en-us/learn/modules/perform-advanced-streaming-data-transformations-with-spark-kafka/6-exercise-provision-hdinsight-perform-data-transformations
https://learn.microsoft.com/en-us/samples/azure-samples/hdinsight-spark-scala-kafka-cosmosdb/use-apache-kafka-with-apache-spark-on-hdinsight/#:~:text=NOTE%3A%20Apache%20Kafka%20and%20Spark,cluster%20on%20the%20virtual%20network.

In both of them, I get the error "remainingCores is too small" for the Spark Cluster creation.
I even tried to change the VM in the template and still same error.

Answer 1

Hello @Anonymous ,

Welcome to Microsoft Q&A platform.

As per my repro, I have got the same error message. After drill down I have found it is because out of the CPU quotas limit.

You will receive this error message when you reach a CPU quota limit on the selected region.

Example: If you check the template, “Edit template” you will find the two clusters.

Spark cluster: (2 head nodes each with each 4 cores + 4 worker nodes with each 8 cores) => (2x4 cores + 4x8 cores) = 40 cores

Kafka cluster: (2 head nodes each with each 4 cores + 3 worker nodes with each 8 cores + 3 Zookeeper nodes 2 cores) => (2x4 cores + 3x8 cores + 3x2 cores) = 38 cores

To successfully create the clusters, you need total 78 cores in the selected regions.

The only quota limit is the CPU Cores quota that exists at the region level for each subscription. For example, your subscription may have a 30 CPU core limit in the East US region, with another 30 CPU cores allowed in the East US.

To resolve this issue, I would suggest you reduce the size of the worker nodes OR request quota increase in the region.

Steps to: Requesting quota increases for Azure HDInsight.

Hope this helps. Do let us know if you any further queries.

----------------------------------------------------------------------------------------

Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.