Share via

azure data factory pipeline for databricks notebook.

Niren Adhikary 96 Reputation points
Dec 20, 2022, 7:22 PM

Hello,

I am trying to setup a data factory pipeline that will execute multiple data bricks notebooks on ADF in the order like one after another with dependency on the first notebook. At present the notebook runs in parallel. So the idea is if we have multiple notebooks being added it will run in series one after the another.

For better understanding of the pipeline flow. I am sharing below screenshot. I can add another notebook (like notebook 2 in the flow) at the end, basically the user notebooks but when we have multiple notebooks being added then it becomes difficult to keep adding so looking for a way to run them without adding multiple stages or activities.

272546-image.png

Kindly suggest.

Thanks,

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,331 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,259 questions
{count} vote

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,511 Reputation points Microsoft Employee
    Dec 21, 2022, 2:41 PM

    Hi @Niren Adhikary ,

    Thank you for posting query in Microsoft Q&A Platform.

    If I understand correctly, you want to execute multiple notebooks in pipeline. Please correct me if I am wrong.

    To execute multiple notebooks, we should consider adding all of them using notebook activity. For example, If we want to run 3 different notebooks, then 3 notebook activities in pipeline.

    OR

    If all your notebooks are in same workspace, then you can make save all notebook paths in some file or SQL table and read them using lookup activity and then loop that notebooks names using ForEach activity and inside Foreach activity use only one notebook activity. In notebook activity under settings tab use notebook path dynamic expression and pass each notebook name for each execution.

    Hope this helps.

    -------------

    Please consider hitting Accept Answer button. Accepted answers help community as well.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.