New-AzSynapseSparkPool
Creates a Synapse Analytics Spark pool.
Syntax
CreateByNameAndEnableAutoScaleParameterSet (Default)
New-AzSynapseSparkPool
-WorkspaceName <String>
-Name <String>
-NodeSize <String>
-AutoScaleMinNodeCount <Int32>
-AutoScaleMaxNodeCount <Int32>
-SparkVersion <String>
[-ResourceGroupName <String>]
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
CreateByNameAndDisableAutoScaleParameterSet
New-AzSynapseSparkPool
-WorkspaceName <String>
-Name <String>
-NodeCount <Int32>
-NodeSize <String>
-SparkVersion <String>
[-ResourceGroupName <String>]
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
CreateByParentObjectAndEnableAutoScaleParameterSet
New-AzSynapseSparkPool
-WorkspaceObject <PSSynapseWorkspace>
-Name <String>
-NodeSize <String>
-AutoScaleMinNodeCount <Int32>
-AutoScaleMaxNodeCount <Int32>
-SparkVersion <String>
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
CreateByParentObjectAndDisableAutoScaleParameterSet
New-AzSynapseSparkPool
-WorkspaceObject <PSSynapseWorkspace>
-Name <String>
-NodeCount <Int32>
-NodeSize <String>
-SparkVersion <String>
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
Description
The New-AzSynapseSparkPool cmdlet creates an Azure Synapse Analytics Spark pool.
Examples
Example 1
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool.
Example 2
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -AutoScaleMinNodeCount 3 -AutoScaleMaxNodeCount 10 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool with auto-scale enabled.
Example 3
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableDynamicExecutorAllocation -MinExecutorCount 1 -MaxExecutorCount 4 -NodeCount 10 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool with dynamic executor allocation enabled and specify min executor count and max executor count.
Example 4
$config = Get-AzSynapseSparkConfiguration -WorkspaceName ContosoWorkspace -Name ContosoSparkConfig1
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize Small -SparkConfiguration $config
This command creates an Azure Synapse Analytics Spark pool and specify a Spark configuration for Spark pool.
Example 5
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool through pipeline.
Example 6
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -AutoScaleMinNodeCount 3 -AutoScaleMaxNodeCount 10 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool with auto-scale enabled through pipeline.
Example 7
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -EnableIsolatedCompute -NodeSize XXXLarge -NodeCount 3 -SparkVersion 2.4
This command creates an Azure Synapse Analytics Spark pool with isolated compute enabled through pipeline.
Parameters
-AsJob
Run cmdlet in the background
Parameter properties
Type: SwitchParameter
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-AutoPauseDelayInMinute
Number of minutes idle. This parameter can be specified when Auto-pause is enabled. The default value will be [15] if it is not specified manually.
Parameter properties
Type: Int32
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-AutoScaleMaxNodeCount
Maximum number of nodes to be allocated in the specified Spark pool.
This parameter must be specified when Auto-scale is enabled.
Parameter properties
Type: Int32
Default value: None Supports wildcards: False DontShow: False
Parameter sets
CreateByNameAndEnableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
CreateByParentObjectAndEnableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-AutoScaleMinNodeCount
Minimum number of nodes to be allocated in the specified Spark pool.
This parameter must be specified when Auto-scale is enabled.
Parameter properties
Type: Int32
Default value: None Supports wildcards: False DontShow: False
Parameter sets
CreateByNameAndEnableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
CreateByParentObjectAndEnableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-Confirm
Prompts you for confirmation before running the cmdlet.
Parameter properties
Type: SwitchParameter
Default value: None Supports wildcards: False DontShow: False Aliases: cf
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-DefaultProfile
The credentials, account, tenant, and subscription used for communication with Azure.
Parameter properties
Type: IAzureContextContainer
Default value: None Supports wildcards: False DontShow: False Aliases: AzContext, AzureRmContext, AzureCredential
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-EnableAutoPause
Indicates whether Auto-pause should be enabled.
Parameter properties
Type: SwitchParameter
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-EnableDynamicExecutorAllocation
Indicates whether dynamic executor allocation should be enabled.
Parameter properties
Type: SwitchParameter
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-EnableIsolatedCompute
The Isolate Compute option is only available with the XXXLarge (80 vCPU / 504 GB) node size. Enabling this option offers isolation for Apache Spark compute for untrusted services. Isolated compute costs the same as the non-isolated VM of the same size. If you expect to enable Isolated Compute for spark pool, ensure that your Synapse workspace is created in an isolated compute supported region, please refer to this document for more details: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-pool-configurations#isolated-compute .
Parameter properties
Type: SwitchParameter
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-MaxExecutorCount
Maximum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maximumNodeCount (exclusive). If it is not specified manually, the default value will be 2.
Parameter properties
Type: Int32
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-MinExecutorCount
Minimum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maxExecutors (exclusive). If it is not specified manually, the default value will be 1.
Parameter properties
Type: Int32
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-Name
Name of Synapse Spark pool.
Parameter properties
Type: String
Default value: None Supports wildcards: False DontShow: False Aliases: SparkPoolName
Parameter sets
(All)
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-NodeCount
Number of nodes to be allocated in the specified Spark pool.
Parameter properties
Type: Int32
Default value: None Supports wildcards: False DontShow: False
Parameter sets
CreateByNameAndDisableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
CreateByParentObjectAndDisableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-NodeSize
Number of core and memory to be used for nodes allocated in the specified Spark pool.
This parameter must be specified when Auto-scale is disabled
Parameter properties
Type: String
Default value: None Accepted values: Small, Medium, Large, XLarge, XXLarge, XXXLarge Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-ResourceGroupName
Resource group name.
Parameter properties
Type: String
Default value: None Supports wildcards: False DontShow: False
Parameter sets
CreateByNameAndEnableAutoScaleParameterSet
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
CreateByNameAndDisableAutoScaleParameterSet
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-SparkConfiguration
Apache Spark configuration. When a job is submitted to the pool, the properties specified in the selected configuration will be referenced.
Parameter properties
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-SparkVersion
Apache Spark version.
Allowed values: 2.4
Parameter properties
Type: String
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-Tag
A string,string dictionary of tags associated with the resource.
Parameter properties
Type: Hashtable
Default value: None Supports wildcards: False DontShow: False
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-WhatIf
Shows what would happen if the cmdlet runs.
The cmdlet is not run.
Parameter properties
Type: SwitchParameter
Default value: None Supports wildcards: False DontShow: False Aliases: wi
Parameter sets
(All)
Position: Named Mandatory: False Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-WorkspaceName
Name of Synapse workspace.
Parameter properties
Type: String
Default value: None Supports wildcards: False DontShow: False
Parameter sets
CreateByNameAndEnableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
CreateByNameAndDisableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: False Value from pipeline by property name: False Value from remaining arguments: False
-WorkspaceObject
workspace input object, usually passed through the pipeline.
Parameter properties
Parameter sets
CreateByParentObjectAndEnableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: True Value from pipeline by property name: False Value from remaining arguments: False
CreateByParentObjectAndDisableAutoScaleParameterSet
Position: Named Mandatory: True Value from pipeline: True Value from pipeline by property name: False Value from remaining arguments: False
CommonParameters
This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable,
-InformationAction, -InformationVariable, -OutBuffer, -OutVariable, -PipelineVariable,
-ProgressAction, -Verbose, -WarningAction, and -WarningVariable. For more information, see
about_CommonParameters .
Outputs