Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Issue published date: October 13, 2023
When you're submitting cluster creation requests by using Azure Resource Manager templates (ARM templates), runbooks, PowerShell, the Azure CLI, and other automation tools, you might receive a "BadRequest" error message if you specify the full component version, for example: clusterType = "Kafka", HDI version = "5.0", and Kafka version = "2.4.1".
Troubleshooting steps
When you're using templates or automation tools to create HDInsight clusters, choose one of the below versions (Major.minor) as componentVersion:
| Cluster type | HDInsight 4.0 | HDInsight 5.0 | HDInsight 5.1 |
|---|---|---|---|
| Hadoop | 3.1 | - | 3.3 |
| Spark | 2.4 | 3.1 | 3.3 |
| Kafka | 2.1 | 2.4 | 3.2 |
| HBase | 2.1 | - | 2.4 |
| InteractiveHive | 3.1 | 3.1 | 3.1 |
This value enables you to successfully create HDInsight clusters. The following snippet shows how to add the component version in the template:
"clusterDefinition": { "kind": "[parameters('clusterKind')]", "componentVersion": { "Hadoop": "3.1" },
Note
Spark 2.4 has reached EOL and is no longer under support by Microsoft. HDInsight versions 4.0 and 5.0 are under basic support. Migrate to HDInsight 5.1 by 31 March 2025.