Add-AzHDInsightConfigValue

Module:

Az.HDInsight Module

Adds a Hadoop configuration value customization and/or a Hive shared library customization to a cluster configuration object.

Syntax

Default (Default)

Add-AzHDInsightConfigValue
    [-Config] <AzureHDInsightConfig>
    [-Core <Hashtable>]
    [-HiveSite <Hashtable>]
    [-HiveEnv <Hashtable>]
    [-OozieSite <Hashtable>]
    [-OozieEnv <Hashtable>]
    [-WebHCat <Hashtable>]
    [-HBaseSite <Hashtable>]
    [-HBaseEnv <Hashtable>]
    [-Storm <Hashtable>]
    [-Yarn <Hashtable>]
    [-MapRed <Hashtable>]
    [-Tez <Hashtable>]
    [-Hdfs <Hashtable>]
    [-RServer <Hashtable>]
    [-SparkDefaults <Hashtable>]
    [-SparkThriftConf <Hashtable>]
    [-Spark2Defaults <Hashtable>]
    [-Spark2ThriftConf <Hashtable>]
    [-DefaultProfile <IAzureContextContainer>]
    [<CommonParameters>]

Description

The Add-AzHDInsightConfigValue cmdlet adds a Hadoop configuration value customization, such as core-site.xml or hive-site.xml, and/or a Hive shared library customization to the HDInsight configuration object created by the New-AzHDInsightClusterConfig cmdlet.

Examples

Example 1: Add a custom configuration value to the cluster configuration object

# Primary storage account info
$storageAccountResourceGroupName = "Group"
$storageAccountResourceId = "yourstorageaccountresourceid"
$storageAccountName = "yourstorageacct001"
$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value

$storageContainer = "container001"

# Cluster configuration info
$location = "East US 2"
$clusterResourceGroupName = "Group"
$clusterName = "your-hadoop-001"
$clusterCreds = Get-Credential

# If the cluster's resource group doesn't exist yet, run:
#   New-AzResourceGroup -Name $clusterResourceGroupName -Location $location

# Config values
$coreConfigs = @{"io.file.buffer.size"="300000"}
$mapRedConfigs = @{"mapred.map.max.attempts"="2"}

# Create the cluster
New-AzHDInsightClusterConfig `
            | Add-AzHDInsightConfigValue `
                -Core $coreConfigs `
                -MapRed $mapRedConfigs `
            | New-AzHDInsightCluster `
                -ClusterType Hadoop `
                -OSType Windows `
                -ClusterSizeInNodes 4 `
                -ResourceGroupName $clusterResourceGroupName `
                -ClusterName $clusterName `
                -HttpCredential $clusterCreds `
                -Location $location `
                -StorageAccountResourceId $storageAccountResourceId `
                -StorageAccountKey $storageAccountKey `
                -StorageContainer $storageAccountContainer

This command adds a Hadoop configuration value to the cluster named your-hadoop-001.

Parameters

-Config

Specifies the HDInsight cluster configuration object that this cmdlet modifies. This object is created by the New-AzHDInsightClusterConfig cmdlet.

Parameter properties

Type:	AzureHDInsightConfig
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	0
Mandatory:	True
Value from pipeline:	True
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Core

Specifies the Core Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Parameter properties

Type:	IAzureContextContainer
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	AzContext, AzureRmContext, AzureCredential

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-HBaseEnv

Specifies the HBase Env configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-HBaseSite

Specifies the HBase Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Hdfs

Specifies the HDFS configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-HiveEnv

Specifies the Hive Env configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-HiveSite

Specifies the Hive Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-MapRed

Specifies the MapRed Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-OozieEnv

Specifies the Oozie Env configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-OozieSite

Specifies the Oozie Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-RServer

Specifies the RServer configurations. Valid only for RServer clusters.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Spark2Defaults

Specifies the Spark2 Defaults configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Spark2ThriftConf

Specifies the Spark2 Thrift SparkConf configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SparkDefaults

Specifies the Spark Defaults configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SparkThriftConf

Specifies the Spark Thrift SparkConf configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Storm

Specifies the Storm Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Tez

Specifies the Tez Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-WebHCat

Specifies the WebHCat Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Yarn

Specifies the YARN Site configurations of this HDInsight cluster.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

CommonParameters

This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutBuffer, -OutVariable, -PipelineVariable, -ProgressAction, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters.

Inputs

AzureHDInsightConfig

Outputs

AzureHDInsightConfig

Related Links

• New-AzHDInsightClusterConfig