Submit-AzSynapseSparkJob

Module:

Az.Synapse Module

Submits a Synapse Analytics Spark job.

Syntax

RunSparkJobParameterSetName (Default)

Submit-AzSynapseSparkJob
    -WorkspaceName <String>
    -SparkPoolName <String>
    -Language <String>
    -Name <String>
    -MainDefinitionFile <String>
    -ExecutorCount <Int32>
    -ExecutorSize <String>
    [-MainClassName <String>]
    [-CommandLineArgument <String[]>]
    [-ReferenceFile <String[]>]
    [-Configuration <Hashtable>]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

RunSparkJobByParentObjectParameterSet

Submit-AzSynapseSparkJob
    -SparkPoolObject <PSSynapseSparkPool>
    -Language <String>
    -Name <String>
    -MainDefinitionFile <String>
    -ExecutorCount <Int32>
    -ExecutorSize <String>
    [-MainClassName <String>]
    [-CommandLineArgument <String[]>]
    [-ReferenceFile <String[]>]
    [-Configuration <Hashtable>]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

Description

The Submit-AzSynapseSparkJob cmdlet submits a Synapse Analytics Spark job.

Examples

Example 1

Submit-AzSynapseSparkJob -WorkspaceName ContosoWorkspace -SparkPoolName ContosoSparkPool -Language Spark -Name WordCount_Java -MainDefinitionFile abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/java/wordcount/wordcount.jar -MainClassName WordCount -CommandLineArgument abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/java/wordcount/shakespeare.txt,abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/java/wordcount/result/ -ExecutorCount 2 -ExecutorSize Small

This command submits a Synapse Analytics Spark job.

Example 2

Submit-AzSynapseSparkJob -WorkspaceName ContosoWorkspace -SparkPoolName ContosoSparkPool -Language SparkDotNet -Name WordCount_Dotnet -MainDefinitionFile abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/dotnet/wordcount/wordcount.zip -MainExecutableFile WordCount -CommandLineArgument abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/dotnet/wordcount/shakespeare.txt,abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/dotnet/wordcount/result -ExecutorCount 2 -ExecutorSize Small

This command submits a Synapse Analytics Spark .NET job.

Example 3

Submit-AzSynapseSparkJob -WorkspaceName ContosoWorkspace -SparkPoolName ContosoSparkPool -Language PySpark -Name WordCount_Python -MainDefinitionFile abfss://ContosoFileSystem@ContosoGen2Storage.blob.core.windows.net/samples/python/wordcount/wordcount.py -CommandLineArgument abfss://ContosoFileSystem@ContosoGen2Storage.blob.core.windows.net/samples/python/wordcount/shakespeare.txt,abfss://ContosoFileSystem@ContosoGen2Storage.blob.core.windows.net/samples/python/wordcount/result/ -ExecutorCount 2 -ExecutorSize Small

This command submits a Synapse Analytics PySpark job.

Parameters

-CommandLineArgument

Optional arguments to the job. e.g. "--iteration 10000 --timeout 20s"

Parameter properties

Type:	String[]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Configuration

Spark configuration properties.

Parameter properties

Type:	Hashtable
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Confirm

Prompts you for confirmation before running the cmdlet.

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	cf

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with Azure.

Parameter properties

Type:	IAzureContextContainer
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	AzContext, AzureRmContext, AzureCredential

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ExecutorCount

Number of executors to be allocated in the specified Spark pool for the job.

Parameter properties

Type:	Int32
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ExecutorSize

Number of core and memory to be used for executors allocated in the specified Spark pool for the job.

Parameter properties

Type:	String
Default value:	None
Accepted values:	Small, Medium, Large, XLarge, XXLarge, XXXLarge
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Language

Language of the job to submit.

Parameter properties

Type:	String
Default value:	None
Accepted values:	Spark, Scala, PySpark, Python, SparkDotNet, CSharp
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-MainClassName

The fully-qualified identifier or the main class that is in the main definition file. Required for Spark and .NET Spark job. e.g. "org.apache.spark.examples.SparkPi"

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	MainExecutableFile

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-MainDefinitionFile

The main file used for the job. e.g. "abfss://filesystem@account.dfs.core.windows.net/mySpark.jar"

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-Name

Name of Spark job.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-ReferenceFile

Additional files used for reference in the main definition file. Comma-separated storage URI list. e.g. "abfss://filesystem@account.dfs.core.windows.net/file1.txt,abfss://filesystem@account.dfs.core.windows.net/result/"

Parameter properties

Type:	String[]
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SparkPoolName

Name of Synapse Spark pool.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

RunSparkJobParameterSetName

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-SparkPoolObject

Spark pool input object, usually passed through the pipeline.

Parameter properties

Type:	PSSynapseSparkPool
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

RunSparkJobByParentObjectParameterSet

Position:	Named
Mandatory:	True
Value from pipeline:	True
Value from pipeline by property name:	False
Value from remaining arguments:	False

-WhatIf

Shows what would happen if the cmdlet runs. The cmdlet is not run.

Parameter properties

Type:	SwitchParameter
Default value:	None
Supports wildcards:	False
DontShow:	False
Aliases:	wi

Parameter sets

(All)

Position:	Named
Mandatory:	False
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

-WorkspaceName

Name of Synapse workspace.

Parameter properties

Type:	String
Default value:	None
Supports wildcards:	False
DontShow:	False

Parameter sets

RunSparkJobParameterSetName

Position:	Named
Mandatory:	True
Value from pipeline:	False
Value from pipeline by property name:	False
Value from remaining arguments:	False

CommonParameters

This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutBuffer, -OutVariable, -PipelineVariable, -ProgressAction, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters.

Inputs

PSSynapseSparkPool

Outputs

PSSynapseSparkJob