Azure Databricks and Multi-Task Jobs and Notebook activity

Question

All,

We have lot of tables to load for which we plan to use notebook activity. We will read the set of tables call the notebook in a loop each time passing the next table in the loop. While the notebook activity method gives you the looping method it also has the disadvantage of running each call in a new jobcluster.

Comparatively, Multi-Task Job seems to offer an alternative which enables you to run many tasks in the same cluster they are also workflow based. Is there a way I can pass a set of parameters that (5 tables for example) once the first run completes (for the 5 tables). If this possible, I can for example, setup 5 tasks multi-task job and keep passing it 5 table names every time one run completes.

Thanks,
Gopi

Answer 1

Hello @Gopinath Rajee ,

Thanks for the question and using MS Q&A platform.

You can use Run Now with Different Parameters to re-run a job with different parameters or different values for existing parameters.

For more details, refer to Azure Databricks - Run a job with different parameters.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators