Output value from spark job

Chris Kingdon 21 Reputation points
2021-12-15T22:36:17.95+00:00

When using spark notebooks there is a way to get an output value that can be used later in the integration pipeline as described here: https://learn.microsoft.com/en-us/azure/synapse-analytics/synapse-notebook-activity?tabs=classical#read-synapse-notebook-cell-output-value.

Is there a way to do something similar with Azure Spark Jobs? I would like to use an id that I generate in the job later in my integration pipeline but I can't find anything on how to go about that.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,115 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 73,651 Reputation points Microsoft Employee
    2021-12-16T12:51:41.183+00:00

    Hello @Chris Kingdon ,

    Thanks for the question and using MS Q&A platform.

    Unfortunately, there is no exitValue property from Spark Job definition activity output.

    In Spark job definition activity, the output will be generated to the seperate folder.

    For more details, refer to Quickstart: Transform data using Apache Spark job definition and Tutorial: Create Apache Spark job definition in Synapse Studio

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful