Manage HDInsight on AKS clusters using Azure REST API
Important
Azure HDInsight on AKS retired on January 31, 2025. Learn more with this announcement.
You need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads.
Important
This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.
Learn how to create an HDInsight cluster using an Azure Resource Manager template and the Azure REST API.
The Azure REST API allows you to perform management operations on services hosted in the Azure platform, including the creation of new resources such as HDInsight clusters.
Create a template
Azure Resource Manager templates are JSON documents that describe a resource group and all resources in it (such as HDInsight on AKS). This template-based approach allows you to define the resources that you need for HDInsight in one template.
Create a cluster
Here we are going to create a cluster in an existing cluster pool.
Variables required in the script
Cluster Name
Cluster Pool Name
Subscription ID
Resource Group Name
Region Name
Cluster Type
SKU
Worker Node count
MSI resource ID:
/subscriptions/<subscription ID>/resourcegroups/<resource group name>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<Managed identity name>MSI client ID
MSI object ID
recommendation-capacity-planning
To create a cluster, copy the following command to your REST API tool.
PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.HDInsight/clusterpools/{clusterPoolName}/clusters/{clusterName}?api-version=2023-06-01-preview
Copy the following template into your REST API body.
{
"location": "East US",
"properties": {
"clusterType": "spark",
"computeProfile": {
"nodes": [
{
"type": "worker",
"vmSize": "Standard_E8ads_v5",
"count": 3
}
]
},
"clusterProfile": {
"clusterVersion": "1.0.6",
"ossVersion": "3.3.1",
"identityProfile": {
"msiResourceId": "/subscriptions/{subscription id}/resourceGroups/{resource group}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{Managed Identity}",
"msiClientId": "{}",
"msiObjectId": "{}"
},
"authorizationProfile": {
"userIds": [
"{Microsoft Entra user id}"
]
},
"serviceConfigsProfiles": [],
"sparkProfile": {
"defaultStorageUrl": "{abfs://<container name>@<storage name>.dfs.core.windows.net/}"
},
"sshProfile": {
"count": 1
}
}
}
}
Run the API call.
Next steps
To Customize and manage the cluster, please refer to the following documentation: Cluster Management using REST API