How to pass Spark Config to Synapse Pipeline Spark Job Definition

Question

Hi,

I need to pass Spark Config options to a Spark Job Definition in Synapse Pipeline. I Added the configuration in the Spark Job definition - when directly submitting the Spark Job from there it appears to work fine, however when I reference the Job definition in my pipeline using Spark Job Definition Object - the config does not seem to be inherited.

There is also no possibility to specifiy the config in the Synapse Pipeline Spark Job definition using the UI, when trying to add it directly in the underlying json my changes are not saved.

I need to add the following parameters for my Spark Job:

spark.driver.userClassPathFirst: "true",
spark.executor.userClassPathFirst: "true"

Answer 1

@Matthias Tauber - Thanks for the question and using MS Q&A platform.

Spark configuration: Specify values for Spark configuration properties listed in the topic: Spark Configuration - Application properties. Users can use default configuration and customized configuration.

To pass Spark Config options to a Spark Job Definition in Synapse Pipeline, you can add the configuration in the "Advanced" section of the Spark Job Definition object in your pipeline. Here are the steps to do this:

When you run your pipeline, the Spark Job Definition object will inherit the Spark Config options you specified in the "Advanced" section.

For more details, refer to Quickstart: Transform data using Apache Spark job definition.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

@PRADEEPCHEEKATLA after some more effort we managed to solve the issue by directly baking the config into the synapse spark job definition object. Thank you for your support.