Azure data factory Databricks for each loop

Question

Hi team,

I have a notebooks that loads multiple tables in database.
If this notebook runs inside for each activity with job cluster, it spins up one cluster for each table.

Can we have a workaround where only one job cluster is spinned up?

Answer 1

Hello @BHAWNA BEDI and welcome to Microsoft Q&A.
It sounds like you want to optimize your Databricks usage by Data Factory. Specifically, minimize cluster creation and maximize re-use.

In the Databricks Linked Service, there are several options when selecting what cluster to use:

New Job Cluster: Make a new cluster every times
Existing Interactive Cluster: Specificy a currently running cluster to re-use.
Existing Instance pool: Draw resources from a pool kept warm in the Databrick service

Of these three, you do not want to use the New Job Cluster, as that is the behavior you are describing; creation of a new cluster for every activity. Either of the other two would do what you want. For production, I'd recomend the Existing Instance pool option. This is because in Databricks you can set your pool to scale up/down as needed, and always have a minimum number of nodes ready.

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Answer 2

Maybe, create a parent notebook and call all child notebooks in it so that you will avoid for loop in ADF and use only single databricks notebook activity?