SynapseSparkJobDefinitionActivity interface

Package:

@azure/arm-datafactory

Execute spark job activity.

Extends

ExecutionActivity

Properties

arguments	User specified arguments to SynapseSparkJobDefinitionActivity.
className	The fully-qualified identifier or the main class that is in the main definition file, which will override the 'className' of the spark job definition you provide. Type: string (or Expression with resultType string).
conf	Spark configuration properties, which will override the 'conf' of the spark job definition you provide.
configurationType	The type of the spark config.
driverSize	Number of core and memory to be used for driver allocated in the specified Spark pool for the job, which will be used for overriding 'driverCores' and 'driverMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).
executorSize	Number of core and memory to be used for executors allocated in the specified Spark pool for the job, which will be used for overriding 'executorCores' and 'executorMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).
file	The main file used for the job, which will override the 'file' of the spark job definition you provide. Type: string (or Expression with resultType string).
files	(Deprecated. Please use pythonCodeReference and filesV2) Additional files used for reference in the main definition file, which will override the 'files' of the spark job definition you provide.
filesV2	Additional files used for reference in the main definition file, which will override the 'jars' and 'files' of the spark job definition you provide.
numExecutors	Number of executors to launch for this job, which will override the 'numExecutors' of the spark job definition you provide. Type: integer (or Expression with resultType integer).
pythonCodeReference	Additional python code files used for reference in the main definition file, which will override the 'pyFiles' of the spark job definition you provide.
scanFolder	Scanning subfolders from the root folder of the main definition file, these files will be added as reference files. The folders named 'jars', 'pyFiles', 'files' or 'archives' will be scanned, and the folders name are case sensitive. Type: boolean (or Expression with resultType boolean).
sparkConfig	Spark configuration property.
sparkJob	Synapse spark job reference.
targetBigDataPool	The name of the big data pool which will be used to execute the spark batch job, which will override the 'targetBigDataPool' of the spark job definition you provide.
targetSparkConfiguration	The spark configuration of the spark job.
type	Polymorphic discriminator, which specifies the different types this object can be

Inherited Properties

dependsOn	Activity depends on condition.
description	Activity description.
linkedServiceName	Linked service reference.
name	Activity name.
onInactiveMarkAs	Status result of the activity when the state is set to Inactive. This is an optional property and if not provided when the activity is inactive, the status will be Succeeded by default.
policy	Activity policy.
state	Activity state. This is an optional property and if not provided, the state will be Active by default.
userProperties	Activity user properties.

Property Details

arguments

User specified arguments to SynapseSparkJobDefinitionActivity.

arguments?: any[]

Property Value

any[]

className

The fully-qualified identifier or the main class that is in the main definition file, which will override the 'className' of the spark job definition you provide. Type: string (or Expression with resultType string).

className?: any

Property Value

any

conf

Spark configuration properties, which will override the 'conf' of the spark job definition you provide.

conf?: any

Property Value

any

configurationType

The type of the spark config.

configurationType?: string

Property Value

string

driverSize

Number of core and memory to be used for driver allocated in the specified Spark pool for the job, which will be used for overriding 'driverCores' and 'driverMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).

driverSize?: any

Property Value

any

executorSize

Number of core and memory to be used for executors allocated in the specified Spark pool for the job, which will be used for overriding 'executorCores' and 'executorMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).

executorSize?: any

Property Value

any

file

The main file used for the job, which will override the 'file' of the spark job definition you provide. Type: string (or Expression with resultType string).

file?: any

Property Value

any

files

(Deprecated. Please use pythonCodeReference and filesV2) Additional files used for reference in the main definition file, which will override the 'files' of the spark job definition you provide.

files?: any[]

Property Value

any[]

filesV2

Additional files used for reference in the main definition file, which will override the 'jars' and 'files' of the spark job definition you provide.

filesV2?: any[]

Property Value

any[]

numExecutors

Number of executors to launch for this job, which will override the 'numExecutors' of the spark job definition you provide. Type: integer (or Expression with resultType integer).

numExecutors?: any

Property Value

any

pythonCodeReference

Additional python code files used for reference in the main definition file, which will override the 'pyFiles' of the spark job definition you provide.

pythonCodeReference?: any[]

Property Value

any[]

scanFolder

Scanning subfolders from the root folder of the main definition file, these files will be added as reference files. The folders named 'jars', 'pyFiles', 'files' or 'archives' will be scanned, and the folders name are case sensitive. Type: boolean (or Expression with resultType boolean).

scanFolder?: any

Property Value

any

sparkConfig

Spark configuration property.

sparkConfig?: {[propertyName: string]: any}

Property Value

{[propertyName: string]: any}

sparkJob

Synapse spark job reference.

sparkJob: SynapseSparkJobReference

Property Value

SynapseSparkJobReference

targetBigDataPool

The name of the big data pool which will be used to execute the spark batch job, which will override the 'targetBigDataPool' of the spark job definition you provide.

targetBigDataPool?: BigDataPoolParametrizationReference

Property Value

BigDataPoolParametrizationReference

targetSparkConfiguration

The spark configuration of the spark job.

targetSparkConfiguration?: SparkConfigurationParametrizationReference

Property Value

SparkConfigurationParametrizationReference

type

Polymorphic discriminator, which specifies the different types this object can be

type: "SparkJob"

Property Value

"SparkJob"

Inherited Property Details

dependsOn

Activity depends on condition.

dependsOn?: ActivityDependency[]

Property Value

ActivityDependency[]

Inherited From ExecutionActivity.dependsOn

description

Activity description.

description?: string

Property Value

string

Inherited From ExecutionActivity.description

linkedServiceName

Linked service reference.

linkedServiceName?: LinkedServiceReference

Property Value

LinkedServiceReference

Inherited From ExecutionActivity.linkedServiceName

name

Activity name.

name: string

Property Value

string

Inherited From ExecutionActivity.name

onInactiveMarkAs

Status result of the activity when the state is set to Inactive. This is an optional property and if not provided when the activity is inactive, the status will be Succeeded by default.

onInactiveMarkAs?: string

Property Value

string

Inherited From ExecutionActivity.onInactiveMarkAs

policy

Activity policy.

policy?: ActivityPolicy

Property Value

ActivityPolicy

Inherited From ExecutionActivity.policy

state

Activity state. This is an optional property and if not provided, the state will be Active by default.

state?: string

Property Value

string

Inherited From ExecutionActivity.state

userProperties

Activity user properties.

userProperties?: UserProperty[]

Property Value

UserProperty[]

Inherited From ExecutionActivity.userProperties