How to run same notebook with different parameters using mssparkutils.notebook.run() in parallel?

Question

I need to run same notebook with different parameters, which should run parallel. Example i have a notebook "NameToPrint", where i need to run this with 5 different parameters ("Ride","Wide","ligth","Special","Look") to reduce run time, i need to run it parallel. How to do it?

Answer 1

Hi @Dinesh Kumar Perumal (Nallas) ,

You can Reference run multiple notebooks in parallel with the help of mssparkutils.notebook.runMultiple() method.

The method mssparkutils.notebook.runMultiple() allows you to run multiple notebooks in parallel or with a predefined topological structure. The API is using a multi-thread implementation mechanism within a spark session, which means the compute resources are shared by the reference notebook runs.

With mssparkutils.notebook.runMultiple(), you can:

• Execute multiple notebooks simultaneously, without waiting for each one to finish.
• Specify the dependencies and order of execution for your notebooks, using a simple JSON format.
• Optimize the use of Spark compute resources and reduce the cost of your Synapse projects.
• View the Snapshots of each notebook run record in the output, and debug/monitor your notebook tasks conveniently.
• Get the exit value of each executive activity and use them in downstream tasks. Alternatively, you could try to run the notebook from another notebook using %run command 5 times with different parameter values in each run command .

 

Or else, call the Notebook within synapse pipeline inside foreach activity by passing array of parameters to loop through in each iterations.

Hope it helps. Kindly let us know how it goes. Thankyou