Azure Synapse notebooks runMultiple() not showing child notebook progress

Question

Azure Synapse notebooks runMultiple() not showing child notebook progress

Lowie Pouwels 0

We are using PySpark notebooks in our Azure Synapse workspace to process our data. Currently we are using belows function to process our notebook in parallel and manage dependencies with a DAG. This works fine, however when one of the notebooks in the DAG fails, we are not getting any good logging.

notebookutils.notebook.runMultiple()

Our output looks like this: User's image We would really like to see the progress and the status of the notebooks that are running or have ran.

When I look at the documentation it should look like this:
User's image

The output above is exactly the output we want. With this output we are able to track the progress and see which notebooks have failed.

We are currently using Azure Synapse and have also tried the mssparkutils.notebook.runMultiple() function. This give the same results

Is there any setting we need to check/enable to get the result and progress in a table like the second screenshot?

J N S S Kasyap 1,520 Reputation points Microsoft External Staff

2025-03-25T18:44:26.5433333+00:00
Hi @Lowie Pouwels

The output in your first screenshot for notebookutils.notebook.runMultiple() shows:

Both notebooks (000_nb_am_generic_functions(0) and (000_nb_am_generic_functions(1)) ran without errors, as exception: None.
However, the exitVal is empty (''), meaning the notebooks did not return any explicit output.

In the second screenshot each notebook (0 and 1) completed successfully.The exitVal field shows a return value ('hello: default value'), indicating that each notebook explicitly returned this value.The exception field is None, meaning no errors were encountered

Your call to notebookutils.notebook.runMultiple() should specify the notebooks explicitly or use a DAG structure.
For simple parallel execution, it’s straightforward:

from notebookutils import notebook result = notebook.runMultiple(["NotebookSimple", "NotebookSimple2"]) print(result)

For a DAG with dependencies, use a dictionary format:

# run multiple notebooks with parameters DAG = { "activities": [ { "name": "NotebookSimple", # activity name, must be unique "path": "NotebookSimple", # notebook path "timeoutPerCellInSeconds": 90, # max timeout for each cell, default to 90 seconds "args": {"p1": "changed value", "p2": 100}, # notebook parameters }, { "name": "NotebookSimple2", "path": "NotebookSimple2", "timeoutPerCellInSeconds": 120, "args": {"p1": "changed value 2", "p2": 200} }, { "name": "NotebookSimple2.2", "path": "NotebookSimple2", "timeoutPerCellInSeconds": 120, "args": {"p1": "changed value 3", "p2": 300}, "retry": 1, "retryIntervalInSeconds": 10, "dependencies": ["NotebookSimple"] # list of activity names that this activity depends on } ], "timeoutInSeconds": 43200, # max timeout for the entire DAG, default to 12 hours "concurrency": 50 # max number of notebooks to run concurrently, default to 50 } notebookutils.notebook.runMultiple(DAG, {"displayDAGViaGraphviz": False})

Ensure the notebook names match exactly (case-sensitive) and are accessible in your Synapse workspace

For better tracking, use:

Manual logging in the parent notebook (print statements).

Explicit exit values in child notebooks (notebook.exit("Success")).

Apache Spark Monitoring in Synapse for execution details.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
J N S S Kasyap 1,520 Reputation points Microsoft External Staff

2025-03-26T16:21:58.17+00:00

@Lowie Pouwels
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 1,520 Reputation points Microsoft External Staff

2025-03-27T19:37:41.1033333+00:00

@Lowie Pouwels
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Your answer

J N S S Kasyap 1,520 Reputation points Microsoft External Staff

2025-03-26T16:21:58.17+00:00

@Lowie Pouwels
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 1,520 Reputation points Microsoft External Staff

2025-03-27T19:37:41.1033333+00:00

@Lowie Pouwels
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Azure Synapse notebooks runMultiple() not showing child notebook progress

Your answer