Edit

Spark Job definition

  • Article

This article provides a breakdown of the structure for Spark job definition items. For detailed information, see How to create and update a Spark Job Definition with Microsoft Fabric Rest API.

Supported formats

SparkJobDefinition items support the SparkJobDefinitionV1 format.

Definition parts

The definition of a Spark job item is made out of a single part, and is constructed as follows:

Example of payload content decoded from Base64

{
    "executableFile": null,
    "defaultLakehouseArtifactId": "",
    "mainClass": "",
    "additionalLakehouseIds": [],
    "retryPolicy": null,
    "commandLineArguments": "",
    "additionalLibraryUris": [],
    "language": "",
    "environmentArtifactId": null
}

Definition example

Here's an example of an item definition for a Spark job.

{
    "format": "SparkJobDefinitionV1",
    "parts": [
        {
            "path": "SparkJobDefinitionV1.json",
            "payload": "eyJleGVjdXRhYmxlRmlsZSI6bnVsbCwiZGVmYXVsdExha2Vob3VzZUFydGlmYWN0SWQiOiIiLCJtYWluQ2xhc3MiOiIiLCJhZGRpdGlvbmFsTGFrZWhvdXNlSWRzIjpbXSwicmV0cnlPbGljYXR5IjpudWxsLCJjb21tYW5kTGluZUFyZ3VtZW50c2I6bnVsbCwiY29tbWFuZExpbmVBYnJndW1lbnRzIjpbXSwibGFuZ3VhZ2UiOiIiLCJlbm52ZW1lbnRBYnJndW1lbnRzIjpbXSwibGFuZ3VhZ2UiOiIiLCJlbm52ZW1lbnRBYnJndW1lbnRzIjpbXSwiZW52aXJvbm1lbnRBcnRpZmFjdElkIjpudWxsfQ==",
            "payloadType": "InlineBase64"
        }
    ]
}