Manage Apache Hadoop clusters in HDInsight by using Azure PowerShell

Artikkeli
10/17/2024

Azure PowerShell can be used to control and automate the deployment and management of your workloads in Azure. In this article, you learn how to manage Apache Hadoop clusters in Azure HDInsight by using the Azure PowerShell Az module. For the list of the HDInsight PowerShell cmdlets, see the Az.HDInsight reference.

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

Note

We recommend that you use the Azure Az PowerShell module to interact with Azure. To get started, see Install Azure PowerShell. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.

The PowerShell Az Module installed.

Create clusters

See Create Linux-based clusters in HDInsight using Azure PowerShell

List clusters

Use the following command to list all clusters in the current subscription:

Get-AzHDInsightCluster

Show cluster

Use the following command to show details of a specific cluster in the current subscription:

Get-AzHDInsightCluster -ClusterName <Cluster Name>

Delete clusters

Use the following command to delete a cluster:

Remove-AzHDInsightCluster -ClusterName <Cluster Name>

You can also delete a cluster by removing the resource group that contains the cluster. Deleting a resource group deletes all the resources in the group including the default storage account.

Remove-AzResourceGroup -Name <Resource Group Name>

Scale clusters

The cluster scaling feature allows you to change the number of worker nodes used by a cluster that is running in Azure HDInsight without having to re-create the cluster. To change the Hadoop cluster size by using Azure PowerShell, run the following command from a client machine:

Set-AzHDInsightClusterSize -ClusterName <Cluster Name> -TargetInstanceCount <NewSize>

For more information about scaling clusters, see Scale HDInsight clusters.

Update HTTP user credentials

Set-AzHDInsightGatewayCredential sets the gateway HTTP credentials of an Azure HDInsight cluster.

$clusterName = "CLUSTERNAME"
$credential = Get-Credential -Message "Enter the HTTP username and password:" -UserName "admin"

Set-AzHDInsightGatewayCredential -ClusterName $clusterName -HttpCredential $credential

Find the default storage account

The following PowerShell script demonstrates how to get the default storage account name and the related information:

#Connect-AzAccount
$clusterName = "<HDInsight Cluster Name>"

$clusterInfo = Get-AzHDInsightCluster -ClusterName $clusterName
$storageInfo = $clusterInfo.DefaultStorageAccount.split('.')
$defaultStoreageType = $storageInfo[1]
$defaultStorageName = $storageInfo[0]

echo "Default Storage account name: $defaultStorageName"
echo "Default Storage account type: $defaultStoreageType"

if ($defaultStoreageType -eq "blob")
{
    $defaultBlobContainerName = $cluster.DefaultStorageContainer
    $defaultStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $defaultStorageAccountName)[0].Value
    $defaultStorageAccountContext = New-AzStorageContext -StorageAccountName $defaultStorageAccountName -StorageAccountKey $defaultStorageAccountKey

    echo "Default Blob container name: $defaultBlobContainerName"
    echo "Default Storage account key: $defaultStorageAccountKey"
}

Find the resource group

In the Resource Manager mode, each HDInsight cluster belongs to an Azure resource group. To find the resource group:

$clusterName = "<HDInsight Cluster Name>"

$cluster = Get-AzHDInsightCluster -ClusterName $clusterName
$resourceGroupName = $cluster.ResourceGroup

Submit jobs

To submit MapReduce jobs

See Run the MapReduce examples included in HDInsight.

To submit Apache Hive jobs

See Run Apache Hive queries using PowerShell.

To submit Apache Sqoop jobs

See Use Apache Sqoop with HDInsight.

To submit Apache Oozie jobs

See Use Apache Oozie with Apache Hadoop to define and run a workflow in HDInsight.

Upload data to Azure Blob storage

See Upload data to HDInsight.

Jaa