New-AzSynapseSparkPool
Creates a Synapse Analytics Spark pool.
Syntax
New-AzSynapseSparkPool
[-ResourceGroupName <String>]
-WorkspaceName <String>
-Name <String>
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
-NodeSize <String>
-AutoScaleMinNodeCount <Int32>
-AutoScaleMaxNodeCount <Int32>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>] New-AzSynapseSparkPool
[-ResourceGroupName <String>]
-WorkspaceName <String>
-Name <String>
[-Tag <Hashtable>]
-NodeCount <Int32>
[-EnableIsolatedCompute]
-NodeSize <String>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>] New-AzSynapseSparkPool
-WorkspaceObject <PSSynapseWorkspace>
-Name <String>
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
-NodeSize <String>
-AutoScaleMinNodeCount <Int32>
-AutoScaleMaxNodeCount <Int32>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>] New-AzSynapseSparkPool
-WorkspaceObject <PSSynapseWorkspace>
-Name <String>
[-Tag <Hashtable>]
-NodeCount <Int32>
[-EnableIsolatedCompute]
-NodeSize <String>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>] Description
The New-AzSynapseSparkPool cmdlet creates an Azure Synapse Analytics Spark pool.
Examples
Example 1
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize SmallThis command creates an Azure Synapse Analytics Spark pool.
Example 2
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -AutoScaleMinNodeCount 3 -AutoScaleMaxNodeCount 10 -SparkVersion 2.4 -NodeSize SmallThis command creates an Azure Synapse Analytics Spark pool with auto-scale enabled.
Example 3
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableDynamicExecutorAllocation -MinExecutorCount 1 -MaxExecutorCount 4 -NodeCount 10 -SparkVersion 2.4 -NodeSize SmallThis command creates an Azure Synapse Analytics Spark pool with dynamic executor allocation enabled and specify min executor count and max executor count.
Example 4
$config = Get-AzSynapseSparkConfiguration -WorkspaceName ContosoWorkspace -Name ContosoSparkConfig1
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize Small -SparkConfiguration $configThis command creates an Azure Synapse Analytics Spark pool and specify a Spark configuration for Spark pool.
Example 5
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize SmallThis command creates an Azure Synapse Analytics Spark pool through pipeline.
Example 6
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -AutoScaleMinNodeCount 3 -AutoScaleMaxNodeCount 10 -SparkVersion 2.4 -NodeSize SmallThis command creates an Azure Synapse Analytics Spark pool with auto-scale enabled through pipeline.
Example 7
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -EnableIsolatedCompute -NodeSize XXXLarge -NodeCount 3 -SparkVersion 2.4This command creates an Azure Synapse Analytics Spark pool with isolated compute enabled through pipeline.
Parameters
-AsJob
Run cmdlet in the background
| Type: | SwitchParameter |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-AutoPauseDelayInMinute
Number of minutes idle. This parameter can be specified when Auto-pause is enabled. The default value will be [15] if it is not specified manually.
| Type: | Int32 |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-AutoScaleMaxNodeCount
Maximum number of nodes to be allocated in the specified Spark pool. This parameter must be specified when Auto-scale is enabled.
| Type: | Int32 |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-AutoScaleMinNodeCount
Minimum number of nodes to be allocated in the specified Spark pool. This parameter must be specified when Auto-scale is enabled.
| Type: | Int32 |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-Confirm
Prompts you for confirmation before running the cmdlet.
| Type: | SwitchParameter |
| Aliases: | cf |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-DefaultProfile
The credentials, account, tenant, and subscription used for communication with Azure.
| Type: | IAzureContextContainer |
| Aliases: | AzContext, AzureRmContext, AzureCredential |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-EnableAutoPause
Indicates whether Auto-pause should be enabled.
| Type: | SwitchParameter |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-EnableDynamicExecutorAllocation
Indicates whether dynamic executor allocation should be enabled.
| Type: | SwitchParameter |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-EnableIsolatedCompute
The Isolate Compute option is only available with the XXXLarge (80 vCPU / 504 GB) node size. Enabling this option offers isolation for Apache Spark compute for untrusted services. Isolated compute costs the same as the non-isolated VM of the same size. If you expect to enable Isolated Compute for spark pool, ensure that your Synapse workspace is created in an isolated compute supported region, please refer to this document for more details: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-pool-configurations#isolated-compute.
| Type: | SwitchParameter |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-MaxExecutorCount
Maximum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maximumNodeCount (exclusive). If it is not specified manually, the default value will be 2.
| Type: | Int32 |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-MinExecutorCount
Minimum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maxExecutors (exclusive). If it is not specified manually, the default value will be 1.
| Type: | Int32 |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-Name
Name of Synapse Spark pool.
| Type: | String |
| Aliases: | SparkPoolName |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-NodeCount
Number of nodes to be allocated in the specified Spark pool.
| Type: | Int32 |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-NodeSize
Number of core and memory to be used for nodes allocated in the specified Spark pool. This parameter must be specified when Auto-scale is disabled
| Type: | String |
| Accepted values: | Small, Medium, Large, XLarge, XXLarge, XXXLarge |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-ResourceGroupName
Resource group name.
| Type: | String |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-SparkConfiguration
Apache Spark configuration. When a job is submitted to the pool, the properties specified in the selected configuration will be referenced.
| Type: | PSSparkConfigurationResource |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-SparkVersion
Apache Spark version. Allowed values: 2.4
| Type: | String |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-Tag
A string,string dictionary of tags associated with the resource.
| Type: | Hashtable |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-WhatIf
Shows what would happen if the cmdlet runs. The cmdlet is not run.
| Type: | SwitchParameter |
| Aliases: | wi |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-WorkspaceName
Name of Synapse workspace.
| Type: | String |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-WorkspaceObject
workspace input object, usually passed through the pipeline.
| Type: | PSSynapseWorkspace |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | True |
| Accept wildcard characters: | False |
Inputs
Outputs
Azure PowerShell
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for