Edit

Share via

Manage HDInsight on AKS clusters using PowerShell

  • Article

Note

We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.

Only basic support will be available until the retirement date.

Important

This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.

Azure PowerShell is a powerful scripting environment that you can use to control and automate the deployment and management of your workloads in Microsoft Azure. This document provides information about how to create a HDInsight on AKS cluster by using Azure PowerShell. It also includes an example script.

Prerequisites

To create an HDInsight on AKS cluster by using Azure PowerShell, you must complete the following procedures:

Set up Azure Environment with PowerShell

The following script demonstrates how to set up an Azure environment with PowerShell:

  1. Open PowerShell
  2. Copy the following code

    1. Install the module Az.HdInsightOnAks Install-Module -Name Az.HdInsightOnAks

      Screenshot shows install the module HDInsight on AKS.

    2. Log in to Azure account and set the default subscription ID Connect-AzAccount Set-AzContext -Subscription {your subscription ID}

Variables required in script

Create HDInsight On AKS cluster pool

Copy the following code to PowerShell

$clusterPoolName="<your cluster pool name>"; 

$resourceGroupName="<your resource group name>"; 

$location="West US 2"; 

$vmSize="Standard_E4s_v3"

Get the available cluster pool version

$clusterPoolVersion=Get-AzHdInsightOnAksAvailableClusterPoolVersion -Location $location 

Write-Output "Start to create cluster pool..." 

$clusterPoolResult=New-AzHdInsightOnAksClusterPool -Name $clusterPoolName -ResourceGroupName $resourceGroupName -Location $location -VmSize $vmSize -ClusterPoolVersion $clusterPoolVersion.ClusterPoolVersionValue

Write-Output "Created cluster pool with name $($clusterPoolResult.Name) successfully"

Create HDInsight On AKS cluster under existing cluster pool

Here, we are going to create cluster under the cluster pool created in the previous step

Run the following code in PowerShell:

Create Trino Cluster



$clusterPoolName=$“{Cluster Pool Name}”;  

$resourceGroupName=$“{Resource Group Name}”;

$location=“{Region Name}”; 

$clusterType="{Trino}"; 

Get available cluster version based the command Get-AzHdInsightOnAksAvailableClusterVersion

$clusterVersion= (Get-AzHdInsightOnAksAvailableClusterVersion -Location $location | Where-Object {$_.ClusterType -eq $clusterType})[0] 
$msiResourceId="<your user msi resource id>"; 
$msiClientId="<your user msi client id>"; 
$msiObjectId="<your msi object id>"; 
$userId="<your Microsoft Entra user id>";

Create node profile

$vmSize="Standard_D8d_v5"; // {Mention the SKU name} 
$workerCount=5; // {Mention the SKU count} 
$nodeProfile = New-AzHdInsightOnAksNodeProfileObject -Type Worker -Count $workerCount -VMSize $vmSize 
$clusterName="<your cluster name>"; 

Write-Output "Start to create cluster..." 

$clusterResult=New-AzHdInsightOnAksCluster -Name $clusterName ` 

                            -PoolName $clusterPoolName ` 

                            -ResourceGroupName $resourceGroupName ` 

                            -Location $location ` 

                            -ClusterType $clusterType ` 

                            -ClusterVersion $clusterVersion.ClusterVersionValue ` 

                            -OssVersion $clusterVersion.OssVersion ` 

                            -AssignedIdentityResourceId $msiResourceId ` 

                            -AssignedIdentityClientId $msiClientId ` 

                            -AssignedIdentityObjectId $msiObjectId ` 

                            -ComputeProfileNode $nodeProfile ` 

                            -AuthorizationUserId $userId

Write-Output "Created cluster with name $($clusterResult.Name) successfully"

Get the cluster with cluster name:

Get-AzHdInsightOnAksCluster -ResourceGroupName $resourceGroupName -PoolName $clusterPoolName -Name $clusterName

List the clusters under the cluster pool

Get-AzHdInsightOnAksCluster -ResourceGroupName $resourceGroupName -PoolName $clusterPoolName

Delete the cluster with cluster name

Remove-AzHdInsightOnAksCluster -Name $clusterName -PoolName $clusterpoolName -ResourceGroupName $resourceGroupName

Next steps

Now you created an HDInsight on AKS cluster, use the following resources to learn how to work with your cluster.

To customize and manage your HDInsight on AKS cluster, refer the following documentation: