cluster in databricks

Question

hello,

in ADF, the pipeline executes a notebook in azure databricks

the cluster of the databricks has stopped and yet the ADF pipeline when calling the databricks notebook, it completes successfully and notebook runs successfully eventhough the cluster is turned off.

Is this behaviour normal?

because I know in databricks itself if I want to execute a notebook, the cluster has to be started.

thanks

Answer 1

@arkiboys - Thanks for the question and using MS Q&A platform.

Yes, it's normal behavior in Azure Databricks.

When you execute a Databricks notebook using Azure Data Factory, the Databricks cluster is started automatically if it is not already running. This is because Azure Data Factory uses the Databricks REST API to start the cluster before executing the notebook.

However, if the cluster is already running, Azure Data Factory will not stop the cluster after the notebook execution is complete. This is because the cluster may be used by other notebooks or jobs, and stopping the cluster may cause issues for those jobs.

So, in your case, if the Databricks cluster was already running when the Azure Data Factory pipeline executed the notebook, the cluster would not have been stopped after the notebook execution was complete.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.