How to call synapse notebook from another notebook inside a loop

Question

I have created a parent notebook which read metadata from Azure SQL server. In metadata there is list of notebooks. I need to call multiple notebooks from parent notebook. tried using for loop but it is not working.

>     df_Metadata = spark.read \  
>             .format("com.microsoft.sqlserver.jdbc.spark") \  
>             .option("url", url) \  
>             .option("query", query_Metadata) \  
>             .option("user", user) \  
>             .option("password", password).load()  
  
pd_Metadata = df_Metadata.toPandas()  
  
for i in range(len(pd_Metadata)):  
        NotebookName = pd_Metadata.loc[i, "Notebook"]  
        %run NotebookName  
    print(pd_Metadata.loc[i, "ObjectId"], pd_Metadata.loc[i, "EDWDatabase"])  

I throws below error:

pd_Metadata = df_Metadata.toPandas()

Answer 1

Hello @Anonymous ,

Thanks for the question and using MS Q&A platform.

Could you please help us understand what does the dataframe pd_Metadata = df_Metadata.toPandas() contains and what exactly the error message which you are experiencing?

You can use %run <notebook path> magic command to reference another notebook within current notebook's context. All the variables defined in the reference notebook are available in the current notebook. %run magic command supports nested calls but not support recursive calls. You will receive an exception if the statement depth is larger than five.

Example: %run /<path>/Notebook1 { "parameterInt": 1, "parameterFloat": 2.5, "parameterBool": true, "parameterString": "abc" }.

Notebook reference works in both interactive mode and Synapse pipeline.

For more details, refer to Notebook reference.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is howd
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

You should probably use the MS Spark Utilities "notebook.run()" functionality instead of the magic command?

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#notebook-utilities