New-AzSynapseSparkPool
Creates a Synapse Analytics Spark pool.
New-AzSynapseSparkPool
[-ResourceGroupName <String>]
-WorkspaceName <String>
-Name <String>
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
-NodeSize <String>
-AutoScaleMinNodeCount <Int32>
-AutoScaleMaxNodeCount <Int32>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
New-AzSynapseSparkPool
[-ResourceGroupName <String>]
-WorkspaceName <String>
-Name <String>
[-Tag <Hashtable>]
-NodeCount <Int32>
[-EnableIsolatedCompute]
-NodeSize <String>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
New-AzSynapseSparkPool
-WorkspaceObject <PSSynapseWorkspace>
-Name <String>
[-Tag <Hashtable>]
[-EnableIsolatedCompute]
-NodeSize <String>
-AutoScaleMinNodeCount <Int32>
-AutoScaleMaxNodeCount <Int32>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
New-AzSynapseSparkPool
-WorkspaceObject <PSSynapseWorkspace>
-Name <String>
[-Tag <Hashtable>]
-NodeCount <Int32>
[-EnableIsolatedCompute]
-NodeSize <String>
[-EnableAutoPause]
[-AutoPauseDelayInMinute <Int32>]
[-EnableDynamicExecutorAllocation]
[-MinExecutorCount <Int32>]
[-MaxExecutorCount <Int32>]
-SparkVersion <String>
[-SparkConfiguration <PSSparkConfigurationResource>]
[-AsJob]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]
The New-AzSynapseSparkPool cmdlet creates an Azure Synapse Analytics Spark pool.
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool.
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -AutoScaleMinNodeCount 3 -AutoScaleMaxNodeCount 10 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool with auto-scale enabled.
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -EnableDynamicExecutorAllocation -MinExecutorCount 1 -MaxExecutorCount 4 -NodeCount 10 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool with dynamic executor allocation enabled and specify min executor count and max executor count.
$config = Get-AzSynapseSparkConfiguration -WorkspaceName ContosoWorkspace -Name ContosoSparkConfig1
New-AzSynapseSparkPool -WorkspaceName ContosoWorkspace -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize Small -SparkConfiguration $config
This command creates an Azure Synapse Analytics Spark pool and specify a Spark configuration for Spark pool.
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -NodeCount 3 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool through pipeline.
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -AutoScaleMinNodeCount 3 -AutoScaleMaxNodeCount 10 -SparkVersion 2.4 -NodeSize Small
This command creates an Azure Synapse Analytics Spark pool with auto-scale enabled through pipeline.
$ws = Get-AzSynapseWorkspace -Name ContosoWorkspace
$ws | New-AzSynapseSparkPool -Name ContosoSparkPool -EnableIsolatedCompute -NodeSize XXXLarge -NodeCount 3 -SparkVersion 2.4
This command creates an Azure Synapse Analytics Spark pool with isolated compute enabled through pipeline.
Run cmdlet in the background
Type: | SwitchParameter |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Number of minutes idle. This parameter can be specified when Auto-pause is enabled. The default value will be [15] if it is not specified manually.
Type: | Int32 |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Maximum number of nodes to be allocated in the specified Spark pool. This parameter must be specified when Auto-scale is enabled.
Type: | Int32 |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Minimum number of nodes to be allocated in the specified Spark pool. This parameter must be specified when Auto-scale is enabled.
Type: | Int32 |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Prompts you for confirmation before running the cmdlet.
Type: | SwitchParameter |
Aliases: | cf |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
The credentials, account, tenant, and subscription used for communication with Azure.
Type: | IAzureContextContainer |
Aliases: | AzContext, AzureRmContext, AzureCredential |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Indicates whether Auto-pause should be enabled.
Type: | SwitchParameter |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Indicates whether dynamic executor allocation should be enabled.
Type: | SwitchParameter |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
The Isolate Compute option is only available with the XXXLarge (80 vCPU / 504 GB) node size. Enabling this option offers isolation for Apache Spark compute for untrusted services. Isolated compute costs the same as the non-isolated VM of the same size. If you expect to enable Isolated Compute for spark pool, ensure that your Synapse workspace is created in an isolated compute supported region, please refer to this document for more details: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-pool-configurations#isolated-compute.
Type: | SwitchParameter |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Maximum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maximumNodeCount (exclusive). If it is not specified manually, the default value will be 2.
Type: | Int32 |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Minimum number of executors to be allocated in the specified Spark pool. This parameter can be specified when DynamicExecutorAllocation is enabled. The value should lie between 1 (inclusive) and maxExecutors (exclusive). If it is not specified manually, the default value will be 1.
Type: | Int32 |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Name of Synapse Spark pool.
Type: | String |
Aliases: | SparkPoolName |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Number of nodes to be allocated in the specified Spark pool.
Type: | Int32 |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Number of core and memory to be used for nodes allocated in the specified Spark pool. This parameter must be specified when Auto-scale is disabled
Type: | String |
Accepted values: | Small, Medium, Large, XLarge, XXLarge, XXXLarge |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Resource group name.
Type: | String |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Apache Spark configuration. When a job is submitted to the pool, the properties specified in the selected configuration will be referenced.
Type: | PSSparkConfigurationResource |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Apache Spark version. Allowed values: 2.4
Type: | String |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | False |
Accept wildcard characters: | False |
A string,string dictionary of tags associated with the resource.
Type: | Hashtable |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Shows what would happen if the cmdlet runs. The cmdlet is not run.
Type: | SwitchParameter |
Aliases: | wi |
Position: | Named |
Default value: | None |
Required: | False |
Accept pipeline input: | False |
Accept wildcard characters: | False |
Name of Synapse workspace.
Type: | String |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | False |
Accept wildcard characters: | False |
workspace input object, usually passed through the pipeline.
Type: | PSSynapseWorkspace |
Position: | Named |
Default value: | None |
Required: | True |
Accept pipeline input: | True |
Accept wildcard characters: | False |
Azure PowerShell feedback
Azure PowerShell is an open source project. Select a link to provide feedback: