Output value from spark job

Question

When using spark notebooks there is a way to get an output value that can be used later in the integration pipeline as described here: https://learn.microsoft.com/en-us/azure/synapse-analytics/synapse-notebook-activity?tabs=classical#read-synapse-notebook-cell-output-value.

Is there a way to do something similar with Azure Spark Jobs? I would like to use an id that I generate in the job later in my integration pipeline but I can't find anything on how to go about that.

Answer 1

Hello @Chris Kingdon ,

Thanks for the question and using MS Q&A platform.

Unfortunately, there is no exitValue property from Spark Job definition activity output.

In Spark job definition activity, the output will be generated to the seperate folder.

For more details, refer to Quickstart: Transform data using Apache Spark job definition and Tutorial: Create Apache Spark job definition in Synapse Studio

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators