New-AzHDInsightCluster

Module:: Az.HDInsight Module

Creates an Azure HDInsight cluster in the specified resource group for the current subscription.

Syntax

Default (Default)

New-AzHDInsightCluster
    [-Location] <String>
    [-ResourceGroupName] <String>
    [-ClusterName] <String>
    [-ClusterSizeInNodes] <Int32>
    [-HttpCredential] <PSCredential>
    [[-StorageAccountResourceId] <String>]
    [[-StorageAccountKey] <String>]
    [-StorageAccountType <StorageType>]
    [-EnableSecureChannel <Boolean>]
    [-Config <AzureHDInsightConfig>]
    [-OozieMetastore <AzureHDInsightMetastore>]
    [-HiveMetastore <AzureHDInsightMetastore>]
    [-AmbariDatabase <AzureHDInsightMetastore>]
    [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
    [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
    [-StorageContainer <String>]
    [-StorageRootPath <String>]
    [-StorageFileSystem <String>]
    [-Version <String>]
    [-HeadNodeSize <String>]
    [-WorkerNodeSize <String>]
    [-EdgeNodeSize <String>]
    [-KafkaManagementNodeSize <String>]
    [-ZookeeperNodeSize <String>]
    [-ClusterType <String>]
    [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-VirtualNetworkId <String>]
    [-SubnetName <String>]
    [-OSType <String>]
    [-ClusterTier <String>]
    [-SshCredential <PSCredential>]
    [-SshPublicKey <String>]
    [-ObjectId <Guid>]
    [-ApplicationId <Guid>]
    [-CertificatePassword <String>]
    [-AadTenantId <Guid>]
    [-SecurityProfile <AzureHDInsightSecurityProfile>]
    [-DisksPerWorkerNode <Int32>]
    [-MinSupportedTlsVersion <String>]
    [-AssignedIdentity <String>]
    [-StorageAccountManagedIdentity <String>]
    [-EncryptionAlgorithm <String>]
    [-EncryptionKeyName <String>]
    [-EncryptionKeyVersion <String>]
    [-EncryptionVaultUri <String>]
    [-EncryptionInTransit <Boolean>]
    [-EncryptionAtHost <Boolean>]
    [-AutoscaleConfiguration <AzureHDInsightAutoscale>]
    [-EnableIDBroker]
    [-KafkaClientGroupId <String>]
    [-KafkaClientGroupName <String>]
    [-ResourceProviderConnection <String>]
    [-PrivateLink <String>]
    [-PublicIpTagType <String>]
    [-PublicIpTag <String>]
    [-OutboundDependenciesManagedType <String>]
    [-EnableComputeIsolation]
    [-ComputeIsolationHostSku <String>]
    [-Zone <String[]>]
    [-Tag <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-PrivateLinkConfiguration <AzureHDInsightPrivateLinkConfiguration[]>]
    [-DefaultProfile <IAzureContextContainer>]
    [<CommonParameters>]

CertificateFilePath

New-AzHDInsightCluster
    [-Location] <String>
    [-ResourceGroupName] <String>
    [-ClusterName] <String>
    [-ClusterSizeInNodes] <Int32>
    [-HttpCredential] <PSCredential>
    [[-StorageAccountResourceId] <String>]
    [[-StorageAccountKey] <String>]
    [-StorageAccountType <StorageType>]
    [-EnableSecureChannel <Boolean>]
    [-Config <AzureHDInsightConfig>]
    [-OozieMetastore <AzureHDInsightMetastore>]
    [-HiveMetastore <AzureHDInsightMetastore>]
    [-AmbariDatabase <AzureHDInsightMetastore>]
    [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
    [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
    [-StorageContainer <String>]
    [-StorageRootPath <String>]
    [-StorageFileSystem <String>]
    [-Version <String>]
    [-HeadNodeSize <String>]
    [-WorkerNodeSize <String>]
    [-EdgeNodeSize <String>]
    [-KafkaManagementNodeSize <String>]
    [-ZookeeperNodeSize <String>]
    [-ClusterType <String>]
    [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-VirtualNetworkId <String>]
    [-SubnetName <String>]
    [-OSType <String>]
    [-ClusterTier <String>]
    [-SshCredential <PSCredential>]
    [-SshPublicKey <String>]
    [-ObjectId <Guid>]
    [-ApplicationId <Guid>]
    [-CertificateFilePath <String>]
    [-CertificatePassword <String>]
    [-AadTenantId <Guid>]
    [-SecurityProfile <AzureHDInsightSecurityProfile>]
    [-DisksPerWorkerNode <Int32>]
    [-MinSupportedTlsVersion <String>]
    [-AssignedIdentity <String>]
    [-StorageAccountManagedIdentity <String>]
    [-EncryptionAlgorithm <String>]
    [-EncryptionKeyName <String>]
    [-EncryptionKeyVersion <String>]
    [-EncryptionVaultUri <String>]
    [-EncryptionInTransit <Boolean>]
    [-EncryptionAtHost <Boolean>]
    [-AutoscaleConfiguration <AzureHDInsightAutoscale>]
    [-EnableIDBroker]
    [-KafkaClientGroupId <String>]
    [-KafkaClientGroupName <String>]
    [-ResourceProviderConnection <String>]
    [-PrivateLink <String>]
    [-PublicIpTagType <String>]
    [-PublicIpTag <String>]
    [-OutboundDependenciesManagedType <String>]
    [-EnableComputeIsolation]
    [-ComputeIsolationHostSku <String>]
    [-Zone <String[]>]
    [-Tag <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-PrivateLinkConfiguration <AzureHDInsightPrivateLinkConfiguration[]>]
    [-DefaultProfile <IAzureContextContainer>]
    [<CommonParameters>]

CertificateFileContents

New-AzHDInsightCluster
    [-Location] <String>
    [-ResourceGroupName] <String>
    [-ClusterName] <String>
    [-ClusterSizeInNodes] <Int32>
    [-HttpCredential] <PSCredential>
    [[-StorageAccountResourceId] <String>]
    [[-StorageAccountKey] <String>]
    [-StorageAccountType <StorageType>]
    [-EnableSecureChannel <Boolean>]
    [-Config <AzureHDInsightConfig>]
    [-OozieMetastore <AzureHDInsightMetastore>]
    [-HiveMetastore <AzureHDInsightMetastore>]
    [-AmbariDatabase <AzureHDInsightMetastore>]
    [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
    [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
    [-StorageContainer <String>]
    [-StorageRootPath <String>]
    [-StorageFileSystem <String>]
    [-Version <String>]
    [-HeadNodeSize <String>]
    [-WorkerNodeSize <String>]
    [-EdgeNodeSize <String>]
    [-KafkaManagementNodeSize <String>]
    [-ZookeeperNodeSize <String>]
    [-ClusterType <String>]
    [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-VirtualNetworkId <String>]
    [-SubnetName <String>]
    [-OSType <String>]
    [-ClusterTier <String>]
    [-SshCredential <PSCredential>]
    [-SshPublicKey <String>]
    [-ObjectId <Guid>]
    [-ApplicationId <Guid>]
    [-CertificateFileContents <Byte[]>]
    [-CertificatePassword <String>]
    [-AadTenantId <Guid>]
    [-SecurityProfile <AzureHDInsightSecurityProfile>]
    [-DisksPerWorkerNode <Int32>]
    [-MinSupportedTlsVersion <String>]
    [-AssignedIdentity <String>]
    [-StorageAccountManagedIdentity <String>]
    [-EncryptionAlgorithm <String>]
    [-EncryptionKeyName <String>]
    [-EncryptionKeyVersion <String>]
    [-EncryptionVaultUri <String>]
    [-EncryptionInTransit <Boolean>]
    [-EncryptionAtHost <Boolean>]
    [-AutoscaleConfiguration <AzureHDInsightAutoscale>]
    [-EnableIDBroker]
    [-KafkaClientGroupId <String>]
    [-KafkaClientGroupName <String>]
    [-ResourceProviderConnection <String>]
    [-PrivateLink <String>]
    [-PublicIpTagType <String>]
    [-PublicIpTag <String>]
    [-OutboundDependenciesManagedType <String>]
    [-EnableComputeIsolation]
    [-ComputeIsolationHostSku <String>]
    [-Zone <String[]>]
    [-Tag <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
    [-PrivateLinkConfiguration <AzureHDInsightPrivateLinkConfiguration[]>]
    [-DefaultProfile <IAzureContextContainer>]
    [<CommonParameters>]

Description

The New-AzHDInsightCluster creates an Azure HDInsight cluster by using the specified parameters or by using a configuration object that is created by using the New-AzHDInsightClusterConfig cmdlet.

The cmdlet may call below Microsoft Graph API according to input parameters:

GET /servicePrincipals/{id}

Examples

Example 1: Create an Azure HDInsight cluster

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds

This command creates a cluster in the current subscription.

Example 2: Create cluster with customer-managed key disk encryption

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-cmk-cluster"
$clusterCreds = Get-Credential

# Customer-managed Key info
$assignedIdentity = "your-ami-resource-id"
$encryptionKeyName = "new-key"
$encryptionVaultUri = "https://MyKeyVault.vault.azure.net"
$encryptionKeyVersion = "00000000000000000000000000000000"

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Spark `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -AssignedIdentity $assignedIdentity `
    -EncryptionKeyName $encryptionKeyName `
    -EncryptionVaultUri $encryptionVaultUri `
    -EncryptionKeyVersion $encryptionKeyVersion

Example 3: Create an Azure HDInsight cluster which enables encryption in transit

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -EncryptionInTransit $true

Example 4: Create an Azure HDInsight cluster with relay outbound and private link feature

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Virtual network info
$virtualNetworkId="yourvnetresourceid"
$subnetName="yoursubnetname"

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -VirtualNetworkId $virtualNetworkId -SubnetName $subnetName `
    -ResourceProviderConnection Outbound -PrivateLink Enabled

Example 5: Create an Azure HDInsight cluster which enables encryption at host

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -EncryptionAtHost $true

Example 6: Create an Azure HDInsight cluster which enables autoscale.

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Create autoscale configuration
$autoscaleConfiguration=New-AzHDInsightClusterAutoscaleConfiguration `
    -MinWorkerNodeCount 3 -MaxWorkerNodeCount 5

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -AutoscaleConfiguration $autoscaleConfiguration

Example 7: Create an Azure HDInsight cluster with Kafka Rest Proxy.

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Kafka Rest Proxy configuration info
$kafkaClientGroupName = "yourclientgroupname"
$kafkaClientGroupId = "yourclientgroupid"
$kafkaManagementNodeSize = "Standard_D4_v2"
$disksPerWorkerNode = 2

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Kafka `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -KafkaClientGroupId  $kafkaClientGroupId -KafkaClientGroupName $kafkaClientGroupName `
    -KafkaManagementNodeSize $kafkaManagementNodeSize -DisksPerWorkerNode $disksPerWorkerNode

Example 8: Create an Azure HDInsight cluster with Azure Data Lake Gen2 storage.

# Primary storage account info
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageManagedIdentity = "yourstorageusermanagedidentity"
$storageFileSystem = "filesystem01"
$storageAccountType = "AzureDataLakeStorageGen2"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 3 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountManagedIdentity $storageManagedIdentity `
    -StorageFileSystem $storageFileSystem `
    -StorageAccountType $storageAccountType `
    -SshCredential $clusterCreds

Example 9: Create an Azure HDInsight cluster with Enterprise Security Package(ESP), Enable HDInsight ID Broker and using WASB storage.

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountKey = "yourstorageaccountaccesskey"
$storageContainer = "yourcontainer01"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# ESP configuration
$domainResourceId = "your Azure AD Domin Service resource id"
$domainUser = "yourdomainuser"
$domainPassword = ConvertTo-SecureString -String "****" -AsPlainText -Force
$domainCredential = New-Object System.Management.Automation.PSCredential($domainUser, $domainPassword)
$clusterUserGroupDns = "dominusergroup"
$ldapUrls = "ldaps://{your domain name}:636"

$clusterTier = "Premium"
$vnetId = "yourvnetid"
$subnetName = "yoursubnetname"
$assignedIdentity = "your user managed assigned identity resourcee id"

#Create security profile
$config= New-AzHDInsightClusterConfig|Add-AzHDInsightSecurityProfile -DomainResourceId $domainResourceId -DomainUserCredential $domainCredential -LdapsUrls $ldapUrls -ClusterUsersGroupDNs $clusterUserGroupDns

# Create the cluster
New-AzHDInsightCluster `
    -ClusterTier $clusterTier `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 3 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -VirtualNetworkId $vnetId -SubnetName $subnetName `
    -AssignedIdentity $assignedIdentity `
    -SecurityProfile $config.SecurityProfile -EnableIDBroker

Example 10: Create an Azure HDInsight cluster which enables compute isolation.

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential
$workerNodeSize="Standard_E16S_V3" # here is just an example
$headNodeSize="Standard_E8S_V3"
$zookeeperNodeSize="Standard_E2S_V3"

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -WorkerNodeSize $workerNodeSize `
    -HeadNodeSize $headNodeSize `
    -ZookeeperNodeSize $zookeeperNodeSize `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -EnableComputeIsolation

Example 11: Create an Azure HDInsight cluster with private link configuration feature

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Virtual network info
$virtualNetworkId="yourvnetresourceid"
$subnetName="yoursubnetname"

$ipConfigName="ipconfig"
$privateIPAllocationMethod="dynamic" # the only supported IP allocation method for private link IP configuration is dynamic
$subnetId=$vnetId+"/subnets/"+$subnetName
# Create Private IP configuration
$ipConfiguration= New-AzHDInsightIPConfiguration -Name $ipConfigName -PrivateIPAllocationMethod $privateIPAllocationMethod -SubnetId $subnetId -Primary

$privateLinkConfigurationName="plconfig"
$groupId="headnode"
# Create private link configuration
$privateLinkConfiguration= New-AzHDInsightPrivateLinkConfiguration -Name $privateLinkConfigurationName -GroupId $groupId -IPConfiguration $ipConfiguration

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -VirtualNetworkId $virtualNetworkId -SubnetName $subnetName `
    -ResourceProviderConnection Outbound -PrivateLink Enabled -PrivateLinkConfiguration $privateLinkConfiguration

Example 12: Create an Azure HDInsight cluster availability feature

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = Get-AzStorageAccountKey `
    -ResourceGroupName $storageAccountResourceGroupName `
    -Name $storageAccountName | Where-Object {$_.KeyName -eq "key1"} | ForEach-Object{$_.Value}
$storageContainer = "container002"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Virtual network info
$virtualNetworkId="yourvnetresourceid"
$subnetName="yoursubnetname"

$databaseUserName="yourusername"
$databasePassword=ConvertTo-SecureString -String "****" -AsPlainText -Force

$sqlserverCredential=New-Object System.Management.Automation.PSCredential($databaseUserName, $databasePassword)
$sqlserver="yoursqlserver.database.windows.net"
$ambariDatabase="ambaridb"
$hiveDatabase ="hivedb"
$oozieDatabase = "ooziedb"

# availability zone feature requires customer ambari database, hive metatastore and oozie metastore
# Create Ambari database
$config=New-AzHDInsightClusterConfig|Add-AzHDInsightMetastore `
-SqlAzureServerName $sqlserver -DatabaseName $ambariDatabase `
-Credential $sqlserverCredential -MetastoreType AmbariDatabase

# Create Hive metastore
$config=$config|Add-AzHDInsightMetastore `
-SqlAzureServerName $sqlserver -DatabaseName $hiveDatabase `
-Credential $sqlserverCredential -MetastoreType HiveMetastore

# Create Oozie metastore
$config=$config|Add-AzHDInsightMetastore `
-SqlAzureServerName $sqlserver -DatabaseName $oozieDatabase `
-Credential $sqlserverCredential -MetastoreType OozieMetastore

# availability zones
$zones="1"

# Create the cluster
New-AzHDInsightCluster `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 4 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountKey $storageAccountKey `
    -StorageContainer $storageContainer `
    -SshCredential $clusterCreds `
    -VirtualNetworkId $virtualNetworkId -SubnetName $subnetName `
    -AmbariDatabase $config.AmbariDatabase -HiveMetastore $config.HiveMetastore -OozieMetastore $config.OozieMetastore -Zone $zones

Example 13: Create an Azure HDInsight cluster with Enterprise Security Package(ESP) and using Azure Data Lake Gen2 storage.

# Primary storage account info
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageManagedIdentity = "yourstorageusermanagedidentity"
$storageFileSystem = "filesystem01"
$storageAccountType = "AzureDataLakeStorageGen2"
# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-002"
$clusterCreds = Get-Credential
# If the cluster's resource group doesn't exist yet, run:
# New-AzResourceGroup -Name $clusterResourceGroupName -Location $location
# ESP configuration
$domainResourceId = "your Azure AD Domin Service resource id"
$domainUser = "yourdomainuser"
$domainPassword = ConvertTo-SecureString -String "****" -AsPlainText -Force
$domainCredential = New-Object System.Management.Automation.PSCredential($domainUser, $domainPassword)
$clusterUserGroupDns = "dominusergroup"
$ldapUrls = "ldaps://{your domain name}:636"
$clusterTier = "Premium"
$vnetId = "yourvnetid"
$subnetName = "yoursubnetname"
$assignedIdentity = "your user managed assigned identity resourcee id"
#Create security profile
$config= New-AzHDInsightClusterConfig|Add-AzHDInsightSecurityProfile -DomainResourceId $domainResourceId -DomainUserCredential $domainCredential -LdapsUrls $ldapUrls -ClusterUsersGroupDNs $clusterUserGroupDns
# Create the cluster
New-AzHDInsightCluster `
    -ClusterTier $clusterTier `
    -ClusterType Hadoop `
    -ClusterSizeInNodes 3 `
    -ResourceGroupName $clusterResourceGroupName `
    -ClusterName $clusterName `
    -HttpCredential $clusterCreds `
    -Location $location `
    -StorageAccountResourceId $storageAccountResourceId `
    -StorageAccountManagedIdentity $storageManagedIdentity `
    -StorageFileSystem $storageFileSystem `
    -StorageAccountType $storageAccountType `
    -SshCredential $clusterCreds `
    -VirtualNetworkId $vnetId -SubnetName $subnetName `
    -AssignedIdentity $assignedIdentity `
    -SecurityProfile $config.SecurityProfile

Parameters

-AadTenantId

Specifies the Microsoft Entra tenant ID that will be used when accessing Azure Data Lake Store.

Parameter properties

Type:	Guid
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-AdditionalStorageAccounts

Specifies the additional Azure Storage accounts for the cluster. You can alternatively use the Add-AzHDInsightStorage cmdlet.

Parameter properties

Type:	Dictionary<TKey,TValue>[System.String,System.String]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-AmbariDatabase

Gets or sets the database for ambari.

Parameter properties

Type:	AzureHDInsightMetastore
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ApplicationId

Gets or sets the Service Principal Application Id for accessing Azure Data Lake.

Parameter properties

Type:	Guid
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-AssignedIdentity

Gets or sets the assigned identity.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-AutoscaleConfiguration

Gets or sets the autoscale configuration

Parameter properties

Type:	AzureHDInsightAutoscale
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-CertificateFileContents

Specifies file contents of the certificate that will be used when accessing Azure Data Lake Store.

Parameter properties

Type:	Byte[]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

CertificateFileContents

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-CertificateFilePath

Specifies the file path to the certificate that will be used to authenticate as the Service Principal. The cluster will use this when accessing Azure Data Lake Store.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

CertificateFilePath

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-CertificatePassword

Specifies the password for the certificate that will be used to authenticate as the Service Principal. The cluster will use this when accessing Azure Data Lake Store.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ClusterName

Specifies the name of the cluster.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	2
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ClusterSizeInNodes

Specifies the number of Worker nodes for the cluster.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	3
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ClusterTier

Specifies the HDInsight cluster tier. By default, this is Standard. The Premium tier can only be used with Linux clusters, and it enables the use of some new features.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ClusterType

Specifies the type of cluster to create. Options are: Hadoop, HBase, Storm, Spark, INTERACTIVEHIVE, Kafka, and RServer

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ComponentVersion

The New-AzHDInsightCluster creates an Azure HDInsight cluster by using the specified parameters or by using a configuration object that is created by using the New-AzHDInsightClusterConfig cmdlet.

The cmdlet may call below Microsoft Graph API according to input parameters:

GET /servicePrincipals/{id}

Parameter properties

Type:	Dictionary<TKey,TValue>[System.String,System.String]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ComputeIsolationHostSku

Gets or sets the dedicated host sku for compute isolation.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Config

Specifies the cluster object to be used to create the cluster. This object can be created by using the New-AzHDInsightClusterConfig cmdlet.

Parameter properties

Type:	AzureHDInsightConfig
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	True
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Configurations

Specifies the configurations of this HDInsight cluster. You can alternatively use the Add-AzHDInsightConfigValues cmdlet.

Parameter properties

Type:	Dictionary<TKey,TValue>[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Parameter properties

Type:	IAzureContextContainer
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	AzContext, AzureRmContext, AzureCredential

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-DisksPerWorkerNode

Specifies the number of disks for worker node role in the cluster.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EdgeNodeSize

Specifies the size of the virtual machine for the edge node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page. This parameter is valid only for RServer clusters.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EnableComputeIsolation

Enables HDInsight compute isolation feature.

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EnableIDBroker

Enables HDInsight Identity Broker feature.

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EnableSecureChannel

Enable secure channel or not, it's an optional field.

Parameter properties

Type:	Nullable<T>[Boolean]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EncryptionAlgorithm

Gets or sets the encryption algorithm.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EncryptionAtHost

Gets or sets the flag which indicates whether enable encryption at host or not.

Parameter properties

Type:	Nullable<T>[Boolean]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EncryptionInTransit

Gets or sets the flag which indicates whether enable encryption in transit or not.

Parameter properties

Type:	Nullable<T>[Boolean]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EncryptionKeyName

Gets or sets the encryption key name.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EncryptionKeyVersion

Gets or sets the encryption key version.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EncryptionVaultUri

Gets or sets the encryption vault uri.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-HeadNodeSize

Specifies the size of the virtual machine for the Head node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-HiveMetastore

Specifies the SQL Database to store Hive metadata. You can alternatively use the Add-AzHDInsightMetastore cmdlet.

Parameter properties

Type:	AzureHDInsightMetastore
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-HttpCredential

Specifies the cluster login (HTTP) credentials for the cluster.

Parameter properties

Type:	PSCredential
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	4
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-KafkaClientGroupId

Gets or sets the client group id for Kafka Rest Proxy access.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-KafkaClientGroupName

Gets or sets the client group name for Kafka Rest Proxy access.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-KafkaManagementNodeSize

Gets or sets the size of the Kafka Management Node.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Location

Specifies the location for the cluster.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	0
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-MinSupportedTlsVersion

Gets or sets the minimal supported TLS version.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ObjectId

Specifies the Microsoft Entra object ID (a GUID) of the Microsoft Entra service principal that represents the cluster. The cluster will use this when accessing Azure Data Lake Store.

Parameter properties

Type:	Guid
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-OozieMetastore

Specifies the SQL Database to store Oozie metadata. You can alternatively use the Add-AzHDInsightMetastore cmdlet.

Parameter properties

Type:	AzureHDInsightMetastore
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-OSType

Specifies the operating system for the cluster. Options are: Windows, Linux

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-OutboundDependenciesManagedType

A value to describe how the outbound dependencies of a HDInsight cluster are managed. 'Managed' means that the outbound dependencies are managed by the HDInsight service. 'External' means that the outbound dependencies are managed by a customer specific solution.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-PrivateLink

Gets or sets the private link type.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-PrivateLinkConfiguration

Gets or sets the private link configuration.

Parameter properties

Type:	AzureHDInsightPrivateLinkConfiguration[]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-PublicIpTag

Gets or sets value of the IpTag associated with the public IP. Example HDInsight, SQL, Storage etc

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-PublicIpTagType

Gets or sets the ipTag type: Example FirstPartyUsage.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ResourceGroupName

Specifies the name of the resource group.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	1
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ResourceProviderConnection

Gets or sets the resource provider connection type.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ScriptActions

Specifies the script actions to run on the cluster at the end of cluster creation. You can alternatively use Add-AzHDInsightScriptAction.

Parameter properties

Type:	Dictionary<TKey,TValue>[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[AzureHDInsightScriptAction]]
Default value:	None
Accepted values:	HeadNode, WorkerNode, ZookeeperNode, EdgeNode
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SecurityProfile

Specifies the security related properties used to create a secure cluster. You can alternatively use the Add-AzHDInsightSecurityProfile cmdlet.

Parameter properties

Type:	AzureHDInsightSecurityProfile
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SshCredential

Specifies the SSH credential to be used for SSH connections. This is only for Linux clusters.

Parameter properties

Type:	PSCredential
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SshPublicKey

Specifies the public key to be used for SSH connections. This is only for Linux clusters.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-StorageAccountKey

Gets or sets the Storage Account Access Key for the Storage Account.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	6
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-StorageAccountManagedIdentity

Gets or sets the storage account managed identity.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-StorageAccountResourceId

Gets or sets the Storage Resource Id for the Storage Account.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	5
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-StorageAccountType

Gets or sets the type of the storage account.

Parameter properties

Type:	Nullable<T>[StorageType]
Default value:	None
Accepted values:	AzureStorage, AzureDataLakeStore, AzureDataLakeStorageGen2
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-StorageContainer

Gets or sets the StorageContainer name for the default Azure Storage Account

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-StorageFileSystem

Gets or sets the file system for the default Azure Data Lake Storage Gen2 account.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-StorageRootPath

Gets or sets the path to the root of the cluster in the default Data Lake Store Account.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SubnetName

Gets or sets the subnet name for this HDInsight cluster.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Tag

Gets or sets the cluster tags.

Parameter properties

Type:	Dictionary<TKey,TValue>[System.String,System.String]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Version

Specifies the HDI version of the HDInsight cluster.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-VirtualNetworkId

Specifies the ID of the virtual network into which to provision the cluster.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-WorkerNodeSize

Specifies the size of the virtual machine for the Worker node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Zone

Gets or sets the availability zones.

Parameter properties

Type:	String[]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ZookeeperNodeSize

Specifies the size of the virtual machine for the Zookeeper node. Use Get-AzVMSize for acceptable VM sizes, and see HDInsight's pricing page. This parameter is valid only for HBase or Storm clusters.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

CommonParameters

This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutBuffer, -OutVariable, -PipelineVariable, -ProgressAction, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters.

Inputs

AzureHDInsightConfig

Outputs

AzureHDInsightCluster

Notes

Keywords: azure, azurerm, arm, resource, management, manager, hadoop, hdinsight, hd, insight

New-AzHDInsightClusterConfig

Share via

New-AzHDInsightCluster

Syntax

Default (Default)

CertificateFilePath

CertificateFileContents

Description

Examples

Example 1: Create an Azure HDInsight cluster

Example 2: Create cluster with customer-managed key disk encryption

Example 3: Create an Azure HDInsight cluster which enables encryption in transit

Example 4: Create an Azure HDInsight cluster with relay outbound and private link feature

Example 5: Create an Azure HDInsight cluster which enables encryption at host

Example 6: Create an Azure HDInsight cluster which enables autoscale.

Example 7: Create an Azure HDInsight cluster with Kafka Rest Proxy.

Example 8: Create an Azure HDInsight cluster with Azure Data Lake Gen2 storage.

Example 9: Create an Azure HDInsight cluster with Enterprise Security Package(ESP), Enable HDInsight ID Broker and using WASB storage.

Example 10: Create an Azure HDInsight cluster which enables compute isolation.

Example 11: Create an Azure HDInsight cluster with private link configuration feature

Example 12: Create an Azure HDInsight cluster availability feature

Example 13: Create an Azure HDInsight cluster with Enterprise Security Package(ESP) and using Azure Data Lake Gen2 storage.

Parameters

-AadTenantId

Parameter properties

Parameter sets

-AdditionalStorageAccounts

Parameter properties

Parameter sets

-AmbariDatabase

Parameter properties

Parameter sets

-ApplicationId

Parameter properties

Parameter sets

-AssignedIdentity

Parameter properties

Parameter sets

-AutoscaleConfiguration

Parameter properties

Parameter sets

-CertificateFileContents

Parameter properties

Parameter sets

-CertificateFilePath

Parameter properties

Parameter sets

-CertificatePassword

Parameter properties

Parameter sets

-ClusterName

Parameter properties

Parameter sets

-ClusterSizeInNodes

Parameter properties

Parameter sets

-ClusterTier

Parameter properties

Parameter sets

-ClusterType

Parameter properties

Parameter sets

-ComponentVersion

Parameter properties

Parameter sets

-ComputeIsolationHostSku

Parameter properties

Parameter sets

-Config

Parameter properties

Parameter sets

-Configurations

Parameter properties

Parameter sets

-DefaultProfile

Parameter properties

Parameter sets

-DisksPerWorkerNode

Parameter properties

Parameter sets

-EdgeNodeSize