Spark Job Definition - Create Or Update Spark Job Definition

Service:

Synapse

API Version:

2020-12-01

Creates or updates a Spark Job Definition.

PUT {endpoint}/sparkJobDefinitions/{sparkJobDefinitionName}?api-version=2020-12-01

URI Parameters

Name	In	Required	Type	Description
endpoint	path	True	string uri	The workspace development endpoint, for example https://myworkspace.dev.azuresynapse.net.
sparkJobDefinitionName	path	True	string	The spark job definition name.
api-version	query	True	string	The Synapse client API Version.

Request Header

Name	Required	Type	Description
If-Match		string	ETag of the Spark Job Definition entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update.

Request Body

Name	Required	Type	Description
properties	True	SparkJobDefinition	Properties of spark job definition.

Responses

Name	Type	Description
200 OK	SparkJobDefinitionResource	OK.
202 Accepted		Accepted.
Other Status Codes	CloudError	An error response received from the Azure Synapse service.

Examples

SparkJobDefinitions_Create

SparkJobDefinitions_Update

SparkJobDefinitions_Create

Sample Request

HTTP

PUT exampleWorkspace.dev.azuresynapse.net/sparkJobDefinitions/exampleSparkJobDefinition?api-version=2020-12-01

{
  "properties": {
    "description": "A sample spark job definition",
    "targetBigDataPool": {
      "referenceName": "exampleBigDataPool",
      "type": "BigDataPoolReference"
    },
    "requiredSparkVersion": "3.3",
    "jobProperties": {
      "name": "exampleSparkJobDefinition",
      "file": "abfss://test@test.dfs.core.windows.net/artefacts/sample.jar",
      "className": "dev.test.tools.sample.Main",
      "conf": {},
      "args": [
        "exampleArg"
      ],
      "jars": [],
      "pyFiles": [],
      "files": [],
      "archives": [],
      "driverMemory": "28g",
      "driverCores": 4,
      "executorMemory": "28g",
      "executorCores": 4,
      "numExecutors": 2
    }
  }
}

Sample Response

Status code:

202

Date: Sat, 16 Jun 2019 00:37:38 GMT
X-Content-Type-Options: nosniff
x-ms-ratelimit-remaining-subscription-writes: 1194
x-ms-request-id: ce95d6dd-c04d-4b02-b7ad-fe79c9b26df0
x-ms-correlation-request-id: ce95d6dd-c04d-4b02-b7ad-fe79c9b26df0

Status code:

200

Date: Sat, 16 Jun 2018 00:37:41 GMT
X-Content-Type-Options: nosniff
x-ms-ratelimit-remaining-subscription-writes: 1192
x-ms-request-id: e4c589b7-a9fe-4c28-981c-3855ec27d264
x-ms-correlation-request-id: e4c589b7-a9fe-4c28-981c-3855ec27d264

Response Body

{
  "id": "/subscriptions/12345678-1234-1234-1234-12345678abc/resourceGroups/exampleResourceGroup/providers/Microsoft.Synapse/workspaces/exampleWorkspaceName/sparkjobdefinitions/exampleSparkJobDefinition",
  "name": "exampleSparkJobDefinition",
  "type": "Microsoft.Synapse/workspaces/sparkjobdefinitions",
  "properties": {
    "description": "A sample spark job definition",
    "targetBigDataPool": {
      "referenceName": "exampleBigDataPool",
      "type": "BigDataPoolReference"
    },
    "requiredSparkVersion": "3.3",
    "jobProperties": {
      "name": "exampleSparkJobDefinition",
      "file": "abfss://test@test.dfs.core.windows.net/artefacts/sample.jar",
      "className": "dev.test.tools.sample.Main",
      "conf": {},
      "args": [
        "exampleArg"
      ],
      "jars": [],
      "pyFiles": [],
      "files": [],
      "archives": [],
      "driverMemory": "28g",
      "driverCores": 4,
      "executorMemory": "28g",
      "executorCores": 4,
      "numExecutors": 2
    }
  },
  "etag": "0a0069d4-0000-0000-0000-5b245bd50000"
}

SparkJobDefinitions_Update

Sample Request

HTTP

PUT exampleWorkspace.dev.azuresynapse.net/sparkJobDefinitions/exampleSparkJobDefinition?api-version=2020-12-01

{
  "properties": {
    "description": "modified description",
    "targetBigDataPool": {
      "referenceName": "exampleBigDataPool",
      "type": "BigDataPoolReference"
    },
    "requiredSparkVersion": "3.3",
    "jobProperties": {
      "name": "exampleSparkJobDefinition",
      "file": "abfss://test@test.dfs.core.windows.net/artefacts/sample.jar",
      "className": "dev.test.tools.sample.Main",
      "conf": {},
      "args": [
        "exampleArg"
      ],
      "jars": [],
      "pyFiles": [],
      "files": [],
      "archives": [],
      "driverMemory": "28g",
      "driverCores": 4,
      "executorMemory": "28g",
      "executorCores": 4,
      "numExecutors": 2
    }
  }
}

Sample Response

Status code:

202

Date: Sat, 16 Jun 2019 00:37:38 GMT
X-Content-Type-Options: nosniff
x-ms-ratelimit-remaining-subscription-writes: 1194
x-ms-request-id: ce95d6dd-c04d-4b02-b7ad-fe79c9b26df0
x-ms-correlation-request-id: ce95d6dd-c04d-4b02-b7ad-fe79c9b26df0

Status code:

200

Date: Sat, 16 Jun 2018 00:37:41 GMT
X-Content-Type-Options: nosniff
x-ms-ratelimit-remaining-subscription-writes: 1192
x-ms-request-id: e4c589b7-a9fe-4c28-981c-3855ec27d264
x-ms-correlation-request-id: e4c589b7-a9fe-4c28-981c-3855ec27d264

Response Body

{
  "id": "/subscriptions/12345678-1234-1234-1234-12345678abc/resourceGroups/exampleResourceGroup/providers/Microsoft.Synapse/workspaces/exampleWorkspaceName/sparkjobdefinitions/exampleSparkJobDefinition",
  "name": "exampleSparkJobDefinition",
  "type": "Microsoft.Synapse/workspaces/sparkjobdefinitions",
  "properties": {
    "description": "modified description",
    "targetBigDataPool": {
      "referenceName": "exampleBigDataPool",
      "type": "BigDataPoolReference"
    },
    "requiredSparkVersion": "3.3",
    "jobProperties": {
      "name": "exampleSparkJobDefinition",
      "file": "abfss://test@test.dfs.core.windows.net/artefacts/sample.jar",
      "className": "dev.test.tools.sample.Main",
      "conf": {},
      "args": [
        "exampleArg"
      ],
      "jars": [],
      "pyFiles": [],
      "files": [],
      "archives": [],
      "driverMemory": "28g",
      "driverCores": 4,
      "executorMemory": "28g",
      "executorCores": 4,
      "numExecutors": 2
    }
  },
  "etag": "0a006cd4-0000-0000-0000-5b245bd60000"
}

Definitions

Name	Description
BigDataPoolReference	Big data pool reference.
BigDataPoolReferenceType	Big data pool reference type.
CloudError	The object that defines the structure of an Azure Synapse error response.
Folder	The folder that this Spark job definition is in. If not specified, this Spark job definition will appear at the root level.
SparkConfigurationReference	Spark configuration reference.
SparkConfigurationReferenceType	Spark configuration reference type.
SparkJobDefinition	Spark job definition.
SparkJobDefinitionResource	Spark job definition resource type.
SparkJobProperties	The properties of the Spark job.

BigDataPoolReference

Big data pool reference.

Name	Type	Description
referenceName	string	Reference big data pool name.
type	BigDataPoolReferenceType	Big data pool reference type.

BigDataPoolReferenceType

Big data pool reference type.

Name	Type	Description
BigDataPoolReference	string

CloudError

The object that defines the structure of an Azure Synapse error response.

Name	Type	Description
error.code	string	Error code.
error.details	CloudError[]	Array with additional error details.
error.message	string	Error message.
error.target	string	Property name/path in request associated with error.

Folder

The folder that this Spark job definition is in. If not specified, this Spark job definition will appear at the root level.

Name	Type	Description
name	string	The name of the folder that this Spark job definition is in.

SparkConfigurationReference

Spark configuration reference.

Name	Type	Description
referenceName	string	Reference spark configuration name.
type	SparkConfigurationReferenceType	Spark configuration reference type.

SparkConfigurationReferenceType

Spark configuration reference type.

Name	Type	Description
SparkConfigurationReference	string

SparkJobDefinition

Spark job definition.

Name	Type	Description
description	string	The description of the Spark job definition.
folder	Folder	The folder that this Spark job definition is in. If not specified, this Spark job definition will appear at the root level.
jobProperties	SparkJobProperties	The properties of the Spark job.
language	string	The language of the Spark application.
requiredSparkVersion	string	The required Spark version of the application.
targetBigDataPool	BigDataPoolReference	Big data pool reference.
targetSparkConfiguration	SparkConfigurationReference	The spark configuration of the spark job.

SparkJobDefinitionResource

Spark job definition resource type.

Name	Type	Description
etag	string	Resource Etag.
id	string	Fully qualified resource ID for the resource. Ex - /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}
name	string	The name of the resource
properties	SparkJobDefinition	Properties of spark job definition.
type	string	The type of the resource. E.g. "Microsoft.Compute/virtualMachines" or "Microsoft.Storage/storageAccounts"

SparkJobProperties

The properties of the Spark job.

Name	Type	Description
archives	string[]	Archives to be used in this job.
args	string[]	Command line arguments for the application.
className	string	Main class for Java/Scala application.
conf	object	Spark configuration properties.
driverCores	integer	Number of cores to use for the driver.
driverMemory	string	Amount of memory to use for the driver process.
executorCores	integer	Number of cores to use for each executor.
executorMemory	string	Amount of memory to use per executor process.
file	string	File containing the application to execute.
files	string[]	files to be used in this job.
jars	string[]	Jars to be used in this job.
name	string	The name of the job.
numExecutors	integer	Number of executors to launch for this job.