Edit

Share via

Manage HDInsight on AKS clusters using Azure REST API

  • Article

Note

We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.

Only basic support will be available until the retirement date.

Important

This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.

Learn how to create an HDInsight cluster using an Azure Resource Manager template and the Azure REST API.

The Azure REST API allows you to perform management operations on services hosted in the Azure platform, including the creation of new resources such as HDInsight clusters.

Create a template

Azure Resource Manager templates are JSON documents that describe a resource group and all resources in it (such as HDInsight on AKS). This template-based approach allows you to define the resources that you need for HDInsight in one template.

Create a cluster

Here we are going to create a cluster in an existing cluster pool.

Variables required in the script

  • Cluster Name

  • Cluster Pool Name

  • Subscription ID

  • Resource Group Name

  • Region Name

  • Cluster Type

  • SKU

  • Worker Node count

  • MSI resource ID:

    /subscriptions/<subscription ID>/resourcegroups/<resource group name>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<Managed identity name>

  • MSI client ID

  • MSI object ID

    Screenshot shows overview.

  • Microsoft Entra user ID

  • HDInsight on AKS VM list

  • recommendation-capacity-planning

To create a cluster, copy the following command to your REST API tool.

PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.HDInsight/clusterpools/{clusterPoolName}/clusters/{clusterName}?api-version=2023-06-01-preview

Copy the following template into your REST API body.

{ 

  "location": "East US", 

  "properties": { 

    "clusterType": "spark", 

    "computeProfile": { 

      "nodes": [ 

        { 

          "type": "worker", 

          "vmSize": "Standard_E8ads_v5", 

          "count": 3 

        } 

      ] 

    }, 

    "clusterProfile": { 

      "clusterVersion": "1.0.6", 

      "ossVersion": "3.3.1", 

      "identityProfile": { 

        "msiResourceId": "/subscriptions/{subscription id}/resourceGroups/{resource group}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{Managed Identity}", 

        "msiClientId": "{}", 

        "msiObjectId": "{}" 

      }, 

      "authorizationProfile": { 

        "userIds": [ 

          "{Microsoft Entra user id}" 

        ] 

      }, 

      "serviceConfigsProfiles": [], 

      "sparkProfile": { 

        "defaultStorageUrl": "{abfs://<container name>@<storage name>.dfs.core.windows.net/}" 

      }, 

      "sshProfile": { 

        "count": 1 

      } 

    } 

  } 

}

Run the API call.

Next steps

To Customize and manage the cluster, please refer to the following documentation: Cluster Management using REST API