How can we parameterise Spark Jobs?

Question

How can we parameterise Spark Jobs?

Steve Homer 36

The Spark Jobs UI in Azure Synapse has the option to pass command line arguments to the underlying code, but there doesn't appear to be any option to parameterise these arguments.

Similarly the Spark Job Definition activity in Azure Pipelines doesn't offer any parameterisation options.

Is there any way to pass parameters to a Azure Synapse job?

PRADEEPCHEEKATLA-MSFT 56,466 Reputation points Microsoft Employee

2020-12-14T09:33:49.143+00:00

Hello @Steve Homer ,

Thanks for the ask and also for using the Microsoft Q&A forum.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.
PRADEEPCHEEKATLA-MSFT 56,466 Reputation points Microsoft Employee

2020-12-16T08:05:45.673+00:00

Hello @Steve Homer ,

We are still waiting for a response from the product team. I will let you know once I heard back from the product team.

Stay Tuned!
David Beavon 896 Reputation points

2022-08-30T17:05:42.573+00:00

In our case we are trying to parameterize the arguments to the spark job, when executed from the REST api:
(sparkJobDefinitions/myjob001/execute)

Any help would be much appreciated. The only alternative seems to be to create lots of similar job definitions that are identical, aside from the arguments.

PRADEEPCHEEKATLA-MSFT 56,466 Reputation points Microsoft Employee

2020-12-14T09:33:49.143+00:00

Hello @Steve Homer ,

Thanks for the ask and also for using the Microsoft Q&A forum.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.
PRADEEPCHEEKATLA-MSFT 56,466 Reputation points Microsoft Employee

2020-12-16T08:05:45.673+00:00

Hello @Steve Homer ,

We are still waiting for a response from the product team. I will let you know once I heard back from the product team.

Stay Tuned!
David Beavon 896 Reputation points

2022-08-30T17:05:42.573+00:00

In our case we are trying to parameterize the arguments to the spark job, when executed from the REST api:
(sparkJobDefinitions/myjob001/execute)

Any help would be much appreciated. The only alternative seems to be to create lots of similar job definitions that are identical, aside from the arguments.

Answer 1

PRADEEPCHEEKATLA-MSFT 56,466 Microsoft Employee

Hello @Steve Homer ,

Currently, the product team working on the public document or tutorial on How can we parameterise Spark jobs.

For now, you can use the job definition JSON file to parameterize the Spark job. Attached one sample file:

{  
  "targetBigDataPool": {  
    "referenceName": "yifso-1019",  
    "type": "SparkComputeReference"  
  },  
  "requiredSparkVersion": "2.4",  
  "jobProperties": {  
    "name": "job definition sample",  
    "file": "wasbs://ContainerName@StorageName.blob.core.windows.net/SparkSubmission/artifact/default_artifact.jar",  
    "className": "sample.LogQuery",  
    "args": [],  
    "jars": [],  
    "pyFiles": [],  
    "archives": [],  
    "files": [],  
    "conf": {  
      "spark.hadoop.fs.azure.account.key.StorageName.blob.core.windows.net": "StorageAccessKey"  
    },  
    "numExecutors": 2,  
    "executorCores": 4,  
    "executorMemory": "14g",  
    "driverCores": 4,  
    "driverMemory": "14g"  
  }  
}

The job definition JSON can be modified, imported, and run directly.

Hope this helps. Do let us know if you any further queries.

------------

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Want a reminder to come back and check responses? Here is how to subscribe to a notification.

PRADEEPCHEEKATLA-MSFT 56,466 Reputation points Microsoft Employee

2020-12-30T11:15:41.873+00:00

Hello @Steve Homer ,

Just checking in to see if the above answer helped. If this answers your query, do click “Accept Answer” and Up-Vote for the same. And, if you have any further query do let us know.
Steve Homer 36 Reputation points

2020-12-30T11:23:26.597+00:00

Hi,

I've only just started to look at this.

As far as I can see this approach would require my using the REST API to submit a separate job for each run with different parameters rather than using the Job Definition in the Synapse UI. Is this correct?

Thanks,
PRADEEPCHEEKATLA-MSFT 56,466 Reputation points Microsoft Employee

2021-01-08T09:31:07.64+00:00

Hello @Steve Homer ,

You can import the job definition JSON file to synapse studio and submit the job definition.

Currently, the document team is working on the tutorial and it will be available soon.

Stay tuned.
David Beavon 896 Reputation points

2022-08-30T17:03:57.583+00:00

@PRADEEPCHEEKATLA-MSFT It has been a year. Is there any way to send parameters when executing a job?

In my case I'd like to use the REST api (sparkJobDefinitions/myjob001/execute)

I don't see any features for parameterization, which requires us to create new job definitions for each run (whenever the argument is changing to my spark driver).

How can we parameterise Spark Jobs?

0 additional answers

Activity