run databricks notebooks parallely

Question

Hi Team,

I need to run multiple notebooks in Azure Data Bricks parallelly using Pyspark . Could you please let me know how to do that?

I am looking forward your response. Thanks.!

Answer 1

Hello @pankaj chaturvedi ,

Thanks for the question and using MS Q&A platform.

You can run multiple notebooks at the same time by using standard Scala and Python constructs such as Threads (Scala, Python) and Futures (Scala, Python). The advanced notebook workflow notebooks demonstrate how to use these constructs. The notebooks are in Scala but you could easily write the equivalent in Python. To run the example:

1. Download the notebook archive.
2. Import the archive into a workspace.
3. Run the Concurrent Notebooks notebook.

For more details, refer to Run multiple notebooks concurrently and Run Databricks Notebooks In Parallel -Python.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a junotification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators