Update-AzSynapseSparkPool

Module:

Az.Synapse Module

Updates a Apache Spark pool in Azure Synapse Analytics.

Syntax

SetByNameParameterSet (Default)

Update-AzSynapseSparkPool
    -WorkspaceName <String>
    -Name <String>
    [-ResourceGroupName <String>]
    [-Tag <Hashtable>]
    [-EnableAutoScale <Boolean>]
    [-AutoScaleMinNodeCount <Int32>]
    [-AutoScaleMaxNodeCount <Int32>]
    [-EnableAutoPause <Boolean>]
    [-AutoPauseDelayInMinute <Int32>]
    [-NodeCount <Int32>]
    [-EnableIsolatedCompute <Boolean>]
    [-NodeSize <String>]
    [-EnableDynamicExecutorAllocation <Boolean>]
    [-MinExecutorCount <Int32>]
    [-MaxExecutorCount <Int32>]
    [-SparkVersion <String>]
    [-LibraryRequirementsFilePath <String>]
    [-SparkConfiguration <PSSparkConfigurationResource>]
    [-PackageAction <PackageActionType>]
    [-Package <System.Collections.Generic.List`1[Microsoft.Azure.Commands.Synapse.Models.WorkspacePackages.PSSynapseWorkspacePackage]>]
    [-ForceApplySetting]
    [-AsJob]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

SetByParentObjectParameterSet

Update-AzSynapseSparkPool
    -Name <String>
    -WorkspaceObject <PSSynapseWorkspace>
    [-Tag <Hashtable>]
    [-EnableAutoScale <Boolean>]
    [-AutoScaleMinNodeCount <Int32>]
    [-AutoScaleMaxNodeCount <Int32>]
    [-EnableAutoPause <Boolean>]
    [-AutoPauseDelayInMinute <Int32>]
    [-NodeCount <Int32>]
    [-EnableIsolatedCompute <Boolean>]
    [-NodeSize <String>]
    [-EnableDynamicExecutorAllocation <Boolean>]
    [-MinExecutorCount <Int32>]
    [-MaxExecutorCount <Int32>]
    [-SparkVersion <String>]
    [-LibraryRequirementsFilePath <String>]
    [-SparkConfiguration <PSSparkConfigurationResource>]
    [-PackageAction <PackageActionType>]
    [-Package <System.Collections.Generic.List`1[Microsoft.Azure.Commands.Synapse.Models.WorkspacePackages.PSSynapseWorkspacePackage]>]
    [-ForceApplySetting]
    [-AsJob]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

SetByInputObjectParameterSet

Update-AzSynapseSparkPool
    -InputObject <PSSynapseSparkPool>
    [-Tag <Hashtable>]
    [-EnableAutoScale <Boolean>]
    [-AutoScaleMinNodeCount <Int32>]
    [-AutoScaleMaxNodeCount <Int32>]
    [-EnableAutoPause <Boolean>]
    [-AutoPauseDelayInMinute <Int32>]
    [-NodeCount <Int32>]
    [-EnableIsolatedCompute <Boolean>]
    [-NodeSize <String>]
    [-EnableDynamicExecutorAllocation <Boolean>]
    [-MinExecutorCount <Int32>]
    [-MaxExecutorCount <Int32>]
    [-SparkVersion <String>]
    [-LibraryRequirementsFilePath <String>]
    [-SparkConfiguration <PSSparkConfigurationResource>]
    [-PackageAction <PackageActionType>]
    [-Package <System.Collections.Generic.List`1[Microsoft.Azure.Commands.Synapse.Models.WorkspacePackages.PSSynapseWorkspacePackage]>]
    [-ForceApplySetting]
    [-AsJob]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

SetByResourceIdParameterSet

Update-AzSynapseSparkPool
    -ResourceId <String>
    [-Tag <Hashtable>]
    [-EnableAutoScale <Boolean>]
    [-AutoScaleMinNodeCount <Int32>]
    [-AutoScaleMaxNodeCount <Int32>]
    [-EnableAutoPause <Boolean>]
    [-AutoPauseDelayInMinute <Int32>]
    [-NodeCount <Int32>]
    [-EnableIsolatedCompute <Boolean>]
    [-NodeSize <String>]
    [-EnableDynamicExecutorAllocation <Boolean>]
    [-MinExecutorCount <Int32>]
    [-MaxExecutorCount <Int32>]
    [-SparkVersion <String>]
    [-LibraryRequirementsFilePath <String>]
    [-SparkConfiguration <PSSparkConfigurationResource>]
    [-PackageAction <PackageActionType>]
    [-Package <System.Collections.Generic.List`1[Microsoft.Azure.Commands.Synapse.Models.WorkspacePackages.PSSynapseWorkspacePackage]>]
    [-ForceApplySetting]
    [-AsJob]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

Description

The Update-AzSynapseSparkPool cmdlet updates an Apache Spark pool in Azure Synapse Analytics.

Note

If the -SparkVersion parameter is used to upgrade the Synapse Spark runtime version, ensure that the Spark pool doesn't have any attached custom libraries or packages. Refer to Migration between Apache Spark versions for more details.

Examples

Example 1

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -Tag @{"key" = "value"} -NodeCount 5 -NodeSize Medium

This command updates an Apache Spark pool in Azure Synapse Analytics.

Example 2

$pool = Get-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool
$pool | Update-AzSynapseSparkPool -Tag @{"key" = "value1"}

This command updates an Apache Spark pool in Azure Synapse Analytics through pipeline.

Example 3

$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | Update-AzSynapseSparkPool -Name ContosoSparkPool -Tag @{"key" = "value2"}

This command updates an Apache Spark pool in Azure Synapse Analytics through pipeline.

Example 4

Update-AzSynapseSparkPool -ResourceId /subscriptions/21686af7-58ec-4f4d-9c68-f431f4db4edd/resourceGroups/ContosoResourceGroup/providers/Microsoft.Synapse/workspaces/ContosoWorkspace/bigDataPools/ContosoSparkPool -Tag @{"key" = "value3"}

This command updates an Apache Spark pool in Azure Synapse Analytics with resource ID.

Example 5

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableAutoScale $true -AutoScaleMinNodeCount 3 -AutoScaleMaxNodeCount 7

This command enables auto-scale for an Apache Spark pool in Azure Synapse Analytics.

Example 6

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableAutoScale $false

This command disables auto-scale for an Apache Spark pool in Azure Synapse Analytics.

Example 7

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableAutoPause $true -AutoPauseDelayInMinute 15

This command enables auto-pause for an Apache Spark pool in Azure Synapse Analytics.

Example 8

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableAutoPause $false

This command disables auto-pause for an Apache Spark pool in Azure Synapse Analytics.

Example 9

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableDynamicExecutorAllocation $true -MinExecutorCount 1 -MaxExecutorCount 5

This command enables dynamic executor allocation and specify min executor count and max executor count for an Apache Spark pool in Azure Synapse Analytics.

Example 10

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableDynamicExecutorAllocation $false

This command disables dynamic executor allocation for an Apache Spark pool in Azure Synapse Analytics.

Example 11

$packages = Get-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace
Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -PackageAction Add -Package $packages

The first command retrieves workspace packages. The second command links these workspace packages to an Apache Spark pool in Azure Synapse Analytics.

Example 12

$package = Get-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace -Name ContosoPackage
Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -PackageAction Remove -Package $package

The first command retrieves workspace packages named ContosoPackage. The second command removes the workspace package from an Apache Spark pool in Azure Synapse Analytics.

Example 13

$pool = Get-AzSynapseSparkPool -ResourceGroupName ContosoResourceGroup -WorkspaceName ContosoWorkspace -Name ContosoSparkPool
$pool | Update-AzSynapseSparkPool -PackageAction Remove -Package $pool.WorkspacePackages

The first command retrieves an Apache Spark pool in Azure Synapse Analytics. The second command removes all workspace packages that are linked to that Apache Spark pool.

Example 14

$workspace_packages = Get-AzSynapseWorkspacePackage -WorkspaceName ContosoWorkspace

$pool = Get-AzSynapseSparkPool -ResourceGroupName ContosoResourceGroup -WorkspaceName ContosoWorkspace -Name ContosoSparkPool
$library_names = $pool.WorkspacePackages | Where-Object {$_.name -notlike "new_package-*"} | ForEach-Object {$_.name}
$library_names += "new_package-2.0-py3-none-any.whl"

$new_pool_packages = @($workspace_packages | Where-Object {$_.name -in $library_names})
Update-AzSynapseSparkPool -ResourceGroupName ContosoResourceGroup -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -PackageAction Set -Package $new_pool_packages

The first command retrieves the packages available in the workspace. The second command group retrieves the spark pool to get the packages currently linked to this pool and removes all versions of the package starting with new_package- from the retrieved list. The new version of the package is then added to this list. In the third group of commands the package list, containing only package names, is tranformed into a list of workspace packages by filtering the list of available workspace_packages accordingly and is then linked to the spark pool.

Example 15

$config = Get-AzSynapseSparkConfiguration -WorkspaceName ContosoWorkspace -Name ContosoSparkConfig1
Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -Tag @{"key" = "value"} -NodeCount 5 -NodeSize Medium -SparkConfiguration $configs

This command updates an Apache Spark pool in Azure Synapse Analytics and specify a Spark configuration for the Spark pool.

Example 16

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -NodeSize small -ForceApplySetting

This command updates an Apache Spark pool in Azure Synapse Analytics, set NodeSize to small for the spark pool and force stop any running jobs in the Spark pool to apply this new setting.

Example 17

$pool = Get-AzSynapseSparkPool -ResourceGroupName ContosoResourceGroup -WorkspaceName ContosoWorkspace -Name ContosoSparkPool
$pool | Update-AzSynapseSparkPool -PackageAction Remove -Package $pool.WorkspacePackages -ForceApplySetting

The first command retrieves an Apache Spark pool in Azure Synapse Analytics. The second command removes all workspace packages that are linked to that Apache Spark pool and force stop any running jobs in the Spark pool to apply this new setting.

Example 18

Update-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableIsolatedCompute $true -NodeSize XXXLarge

This command enables isolated compute and specify node size to XXXLarge(80 vCPU / 504 GB) for an Apache Spark pool in Azure Synapse Analytics.

Parameters

-AsJob

Run cmdlet in the background

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-AutoPauseDelayInMinute

Number of minutes idle. This parameter can be specified when Auto-pause is enabled. The default value will be [15] if it is not specified manually.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-AutoScaleMaxNodeCount

Maximum number of nodes to be allocated in the specified Spark pool. This parameter must be specified when Auto-scale is enabled.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-AutoScaleMinNodeCount

Minimum number of nodes to be allocated in the specified Spark pool. This parameter must be specified when Auto-scale is enabled.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Confirm

Prompts you for confirmation before running the cmdlet.

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	cf

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with Azure.

Parameter properties

Type:	IAzureContextContainer
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	AzContext, AzureRmContext, AzureCredential

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EnableAutoPause

Indicates whether Auto-pause should be enabled.

Parameter properties

Type:	Nullable<T>[Boolean]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EnableAutoScale

Indicates whether Auto-scale should be enabled

Parameter properties

Type:	Nullable<T>[Boolean]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EnableDynamicExecutorAllocation

Indicates whether dynamic executor allocation should be enabled.

Parameter properties

Type:	Nullable<T>[Boolean]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-EnableIsolatedCompute

The Isolate Compute option is only available with the XXXLarge (80 vCPU / 504 GB) node size. Enabling this option offers isolation for Apache Spark compute for untrusted services. Isolated compute costs the same as the non-isolated VM of the same size. If you expect to enable Isolated Compute for spark pool, ensure that your Synapse workspace is created in an isolated compute supported region, please refer to this document for more details: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-pool-configurations#isolated-compute.

Parameter properties

Type:	Nullable<T>[Boolean]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ForceApplySetting

Whether to stop any running jobs in the Big Data pool.

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-InputObject

Spark pool input object, usually passed through the pipeline.

Parameter properties

Type:	PSSynapseSparkPool
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

SetByInputObjectParameterSet

Position:	Named
Mandatory:	True
Value from pipeline:	True
Value from pipeline by property name:	False
Value from remaining arguments:	False

-LibraryRequirementsFilePath

Environment configuration file ("PIP freeze" output).

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-MaxExecutorCount

Maximum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maximumNodeCount (exclusive). If it is not specified manually, the default value will be 2.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-MinExecutorCount

Minimum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maxExecutors (exclusive). If it is not specified manually, the default value will be 1.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Name

Name of Synapse Spark pool.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	SparkPoolName

Parameter sets

SetByNameParameterSet

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

SetByParentObjectParameterSet

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-NodeCount

Number of nodes to be allocated in the specified Spark pool.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-NodeSize

Number of core and memory to be used for nodes allocated in the specified Spark pool. This parameter must be specified when Auto-scale is disabled

Parameter properties

Type:	String
Default value:	None
Accepted values:	Small, Medium, Large, XLarge, XXLarge, XXXLarge
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Package

The workspace packages.

Parameter properties

Type:	List<T>[PSSynapseWorkspacePackage]
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	WorkspacePackage

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-PackageAction

Package action must be specified when you add or remove a workspace package from a Apache Spark pool.

Parameter properties

Type:	Microsoft.Azure.Commands.Synapse.Models.SynapseConstants+PackageActionType
Default value:	None
Accepted values:	Add, Remove, Set
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ResourceGroupName

Resource group name.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

SetByNameParameterSet

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ResourceId

Resource identifier of Synapse Spark pool.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

SetByResourceIdParameterSet

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SparkConfiguration

Apache Spark configuration. When a job is submitted to the pool, the properties specified in the selected configuration will be referenced.

Parameter properties

Type:	PSSparkConfigurationResource
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SparkVersion

Apache Spark version. Allowed values: 3.1,3.2,3.3,3.4

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Tag

A string,string dictionary of tags associated with the resource.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-WhatIf

Shows what would happen if the cmdlet runs. The cmdlet is not run.

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	wi

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-WorkspaceName

Name of Synapse workspace.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

SetByNameParameterSet

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-WorkspaceObject

workspace input object, usually passed through the pipeline.

Parameter properties

Type:	PSSynapseWorkspace
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

SetByParentObjectParameterSet

Position:	Named
Mandatory:	True
Value from pipeline:	True
Value from pipeline by property name:	False
Value from remaining arguments:	False

CommonParameters

This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutBuffer, -OutVariable, -PipelineVariable, -ProgressAction, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters.

Inputs

PSSynapseWorkspace

PSSynapseSparkPool

Outputs

PSSynapseSparkPool