az synapse spark pool

Manage Spark pools.

Commands

Name	Description	Type	Status
az synapse spark pool create	Create a Spark pool.	Core	GA
az synapse spark pool delete	Delete a Spark pool.	Core	GA
az synapse spark pool list	List all Spark pools.	Core	GA
az synapse spark pool show	Get a Spark pool.	Core	GA
az synapse spark pool update	Update the Spark pool.	Core	GA
az synapse spark pool wait	Place the CLI in a waiting state until a condition of a Spark pool is met.	Core	GA

az synapse spark pool create

Edit

Create a Spark pool.

az synapse spark pool create --name
                             --node-count
                             --node-size {Large, Medium, None, Small, XLarge, XXLarge, XXXLarge}
                             --resource-group
                             --spark-version
                             --workspace-name
                             [--delay]
                             [--enable-auto-pause {false, true}]
                             [--enable-auto-scale {false, true}]
                             [--enable-dynamic-exec {false, true}]
                             [--max-executors]
                             [--max-node-count]
                             [--min-executors]
                             [--min-node-count]
                             [--no-wait]
                             [--node-size-family {HardwareAcceleratedFPGA, HardwareAcceleratedGPU, MemoryOptimized, None}]
                             [--spark-config-file-path]
                             [--spark-events-folder]
                             [--spark-log-folder]
                             [--tags]

Examples

Create a Spark pool.

az synapse spark pool create --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--spark-version 2.4 --node-count 3 --node-size Medium --spark-config-file-path 'path/configfile.txt'

Required Parameters

--name -n

The name of the Spark pool.

--node-count

The number of node.

Property	Value
Parameter group:	Node Arguments

--node-size

The level of compute power that each node in the Big Data pool has..

Property	Value
Parameter group:	Node Arguments
Accepted values:	Large, Medium, None, Small, XLarge, XXLarge, XXXLarge

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--spark-version

The supported Spark version is 2.4 now.

Property	Value
Parameter group:	Component Version Arguments

--workspace-name

The workspace name.

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--delay

The delay time whose unit is minute.

Property	Value
Parameter group:	AutoPause Arguments

--enable-auto-pause

The flag of enabling auto pause.

Property	Value
Parameter group:	AutoPause Arguments
Accepted values:	false, true

--enable-auto-scale

The flag of enabling auto scale.

Property	Value
Parameter group:	AutoScale Arguments
Accepted values:	false, true

--enable-dynamic-exec

Indicates whether Dynamic Executor Allocation is enabled or not.

Property	Value
Parameter group:	DynamicExecutor Arguments
Accepted values:	false, true

--max-executors

The maximum number of executors alloted.

Property	Value
Parameter group:	DynamicExecutor Arguments

--max-node-count

The max node count.

Property	Value
Parameter group:	AutoScale Arguments

--min-executors

The minimum number of executors alloted.

Property	Value
Parameter group:	DynamicExecutor Arguments

--min-node-count

The min node count.

Property	Value
Parameter group:	AutoScale Arguments

--no-wait

Do not wait for the long-running operation to finish.

Property	Value
Default value:	False

--node-size-family

The kind of nodes that the Big Data pool provides.

Property	Value
Parameter group:	Node Arguments
Default value:	MemoryOptimized
Accepted values:	HardwareAcceleratedFPGA, HardwareAcceleratedGPU, MemoryOptimized, None

--spark-config-file-path

Absolute path of Spark pool properties configuration file.

Property	Value
Parameter group:	Environment Configuration Arguments

--spark-events-folder

The Spark events folder.

Property	Value
Parameter group:	Default Folder Arguments
Default value:	/events

--spark-log-folder

The default Spark log folder.

Property	Value
Parameter group:	Default Folder Arguments
Default value:	/logs

--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az synapse spark pool delete

Edit

Delete a Spark pool.

az synapse spark pool delete [--ids]
                             [--name]
                             [--no-wait]
                             [--resource-group]
                             [--subscription]
                             [--workspace-name]
                             [--yes]

Examples

Delete a Spark pool.

az synapse spark pool delete --name testpool --workspace-name testsynapseworkspace --resource-group rg

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

Property	Value
Parameter group:	Resource Id Arguments

--name -n

The name of the Spark pool.

Property	Value
Parameter group:	Resource Id Arguments

--no-wait

Do not wait for the long-running operation to finish.

Property	Value
Default value:	False

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Property	Value
Parameter group:	Resource Id Arguments

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Property	Value
Parameter group:	Resource Id Arguments

--workspace-name

The workspace name.

Property	Value
Parameter group:	Resource Id Arguments

--yes -y

Do not prompt for confirmation.

Property	Value
Default value:	False

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az synapse spark pool list

Edit

List all Spark pools.

az synapse spark pool list --resource-group
                           --workspace-name

Examples

List all Spark pools.

az synapse spark pool list --workspace-name testsynapseworkspace --resource-group rg

Required Parameters

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name

The workspace name.

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az synapse spark pool show

Edit

Get a Spark pool.

az synapse spark pool show [--ids]
                           [--name]
                           [--resource-group]
                           [--subscription]
                           [--workspace-name]

Examples

Get a Spark pool.

az synapse spark pool show --name testpool --workspace-name testsynapseworkspace --resource-group rg

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--ids

Property	Value
Parameter group:	Resource Id Arguments

--name -n

The name of the Spark pool.

Property	Value
Parameter group:	Resource Id Arguments

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Property	Value
Parameter group:	Resource Id Arguments

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Property	Value
Parameter group:	Resource Id Arguments

--workspace-name

The workspace name.

Property	Value
Parameter group:	Resource Id Arguments

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az synapse spark pool update

Edit

Update the Spark pool.

az synapse spark pool update [--delay]
                             [--enable-auto-pause {false, true}]
                             [--enable-auto-scale {false, true}]
                             [--enable-dynamic-exec {false, true}]
                             [--force {false, true}]
                             [--ids]
                             [--library-requirements]
                             [--max-executors]
                             [--max-node-count]
                             [--min-executors]
                             [--min-node-count]
                             [--name]
                             [--no-wait]
                             [--node-count]
                             [--node-size {Large, Medium, None, Small, XLarge, XXLarge, XXXLarge}]
                             [--package]
                             [--package-action {Add, Remove}]
                             [--resource-group]
                             [--spark-config-file-path]
                             [--subscription]
                             [--tags]
                             [--workspace-name]

Examples

Update the Spark pool's tags.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--tags key1=value1

Update the Spark pool's auto scale configuration.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--enable-auto-scale --min-node-count 3 --max-node-count 100

Update the Spark pool's custom libraries.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--package-action Add --package package1.jar package2.jar

Update the Spark pool's configuration file.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--spark-config-file-path 'path/configfile.txt'

Update the Spark pool's dynamic executor allocation configuration.

az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--enable-dynamic-exec --min-executors 3 --max-executors 10

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--delay

The delay time whose unit is minute.

Property	Value
Parameter group:	AutoPause Arguments

--enable-auto-pause

The flag of enabling auto pause.

Property	Value
Parameter group:	AutoPause Arguments
Accepted values:	false, true

--enable-auto-scale

The flag of enabling auto scale.

Property	Value
Parameter group:	AutoScale Arguments
Accepted values:	false, true

--enable-dynamic-exec

Indicates whether Dynamic Executor Allocation is enabled or not.

Property	Value
Parameter group:	DynamicExecutor Arguments
Accepted values:	false, true

--force

The flag of force operation.

Property	Value
Default value:	False
Accepted values:	false, true

--ids

Property	Value
Parameter group:	Resource Id Arguments

--library-requirements

The library requirements file.

Property	Value
Parameter group:	Environment Configuration Arguments

--max-executors

The maximum number of executors alloted.

Property	Value
Parameter group:	DynamicExecutor Arguments

--max-node-count

The max node count.

Property	Value
Parameter group:	AutoScale Arguments

--min-executors

The minimum number of executors alloted.

Property	Value
Parameter group:	DynamicExecutor Arguments

--min-node-count

The min node count.

Property	Value
Parameter group:	AutoScale Arguments

--name -n

The name of the Spark pool.

Property	Value
Parameter group:	Resource Id Arguments

--no-wait

Do not wait for the long-running operation to finish.

Property	Value
Default value:	False

--node-count

The number of node.

Property	Value
Parameter group:	Node Arguments

--node-size

The level of compute power that each node in the Big Data pool has..

Property	Value
Parameter group:	Node Arguments
Accepted values:	Large, Medium, None, Small, XLarge, XXLarge, XXXLarge

--package

List of workspace packages name.

Property	Value
Parameter group:	Custom Libraries Arguments

--package-action

Package action must be specified when you add or remove a workspace package from a Apache Spark pool.

Property	Value
Parameter group:	Custom Libraries Arguments
Accepted values:	Add, Remove

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Property	Value
Parameter group:	Resource Id Arguments

--spark-config-file-path

Absolute path of Spark pool properties configuration file.

Property	Value
Parameter group:	Environment Configuration Arguments

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Property	Value
Parameter group:	Resource Id Arguments

--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.

--workspace-name

The workspace name.

Property	Value
Parameter group:	Resource Id Arguments

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az synapse spark pool wait

Edit

Place the CLI in a waiting state until a condition of a Spark pool is met.

az synapse spark pool wait --big-data-pool-name
                           [--created]
                           [--custom]
                           [--deleted]
                           [--exists]
                           [--ids]
                           [--interval]
                           [--resource-group]
                           [--subscription]
                           [--timeout]
                           [--updated]
                           [--workspace-name]

Required Parameters

--big-data-pool-name

Big Data pool name.

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--created

Wait until created with 'provisioningState' at 'Succeeded'.

Property	Value
Parameter group:	Wait Condition Arguments
Default value:	False

--custom

Wait until the condition satisfies a custom JMESPath query. E.g. provisioningState!='InProgress', instanceView.statuses[?code=='PowerState/running'].

Property	Value
Parameter group:	Wait Condition Arguments

--deleted

Wait until deleted.

Property	Value
Parameter group:	Wait Condition Arguments
Default value:	False

--exists

Wait until the resource exists.

Property	Value
Parameter group:	Wait Condition Arguments
Default value:	False

--ids

Property	Value
Parameter group:	Resource Id Arguments

--interval

Polling interval in seconds.

Property	Value
Parameter group:	Wait Condition Arguments
Default value:	30

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Property	Value
Parameter group:	Resource Id Arguments

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Property	Value
Parameter group:	Resource Id Arguments

--timeout

Maximum wait in seconds.

Property	Value
Parameter group:	Wait Condition Arguments
Default value:	3600

--updated

Wait until updated with provisioningState at 'Succeeded'.

Property	Value
Parameter group:	Wait Condition Arguments
Default value:	False

--workspace-name

The workspace name.

Property	Value
Parameter group:	Resource Id Arguments

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

Share via

az synapse spark pool

Commands

az synapse spark pool create

Examples

Required Parameters

Optional Parameters

az synapse spark pool delete

Examples

Optional Parameters

az synapse spark pool list

Examples

Required Parameters

az synapse spark pool show

Examples

Optional Parameters

az synapse spark pool update

Examples

Optional Parameters

az synapse spark pool wait

Required Parameters

Optional Parameters

Feedback