mssparkutils.notebook.exit() doesn't seem to work correctly in pipeline

Question

I have the following Spark Notebook

CDMList = []
DBList = []

%%pyspark

from notebookutils import mssparkutils

set1 = set(CDMList)
set2 = set(DBList)

missing = list(sorted(set1 - set2))
#print(missing)
mssparkutils.notebook.exit(missing)

When I run it by itself it works.

ExitValue: ['LocalAccountCoverage', 'LocalAccountSubject', 'LocalActivityDefinitionObservationRequirement']

I am calling it in a pipeline in Synapse. The Output has exitValue of "[]" when it should have the above. Just to be sure I added a Set variable activity and put in the following:

@activity('CompareLists').output.status.Output.result.exitValue

It sets it to "[]" as well.

Answer 1

Hello @TerriblyVexed ,

To pass parameters, please make sure the cell has marked to parameter cell as shown below.

Try to split the code into two cells and first cell should be marked as toggle parameter cell and modify the code as shown below:

Cell1:[Marked as parameters]

CDMList = ''  
DBList = ''  

Cell2:

from notebookutils import mssparkutils  
set1 = set(CDMList.split(','))  
set2 = set(DBList.split(','))  
missing = list(sorted(set1 - set2))  
#print(missing)  
mssparkutils.notebook.exit(missing)  

Try to remove the single quotes in the parameters.

Note: If you use the parameters with the single quotes the excepted output: "exitValue": "[\"'LocalAccountCoverage'\", \"'LocalAccountSubject'\", \"'LocalActivityDefinitionObservationRequirement'\"]",

Now you will see the excepted output as shown below:

Hope this will help. Please let us know if any further queries.

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of [Q&A Volunteer Moderators][50}

Answer 2

Hello @TerriblyVexed ,

Thanks for the question and using MS Q&A platform.

Your question is not clear enough, because if you look at the Spark Notebook provided above don't have any values in CDMList and DBList and not sure how the it returns the ExitValue: ['LocalAccountCoverage', 'LocalAccountSubject', 'LocalActivityDefinitionObservationRequirement'].

Created a Notebook6 with empty values in CDMList and DBList returns output is Null.

Pipeline associated with Notebook6 which has empty values returns @activity('Notebook1').output.status.Output.result.exitValue is Null.

Created a Notebook7 with values in CDMList returns output is ['LocalAccountCoverage', 'LocalAccountSubject', 'LocalActivityDefinitionObservationRequirement']

Pipeline associated with Notebook7 with values in CDMList returnss @activity('Notebook1').output.status.Output.result.exitValue is ['LocalAccountCoverage', 'LocalAccountSubject', 'LocalActivityDefinitionObservationRequirement']

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators