How to orchestrate pipelines and notebook?

Question

Hi,

What would be the best practice for me to schedule pipelines created in different ADF and notebooks created in Data Bricks.

for example:

Run pipeline1 which is in ADF_A -> Run pileline1 which is in ADF_B -> Run notebook1 which is in Data Bricks

what would be the best form/resource within azure.

Thanks

Answer 1

Hello @Charbel Daia Martins ,

Thanks for the question and using the MS Q&A platform.

There are multiple ways of doing it.

• Using Control flow activity - Execute Pipeline activity allows a Data Factory pipeline to invoke another pipeline.
• Using Activity Dependency defines how subsequent activities depend on previous activities, determining the condition of whether to continue executing the next task. An activity can depend on one or multiple previous activities with different dependency conditions.
• Using Tumbling window trigger dependency - You can create dependent pipelines in your ADF by adding dependencies among tumbling window triggers in your pipelines. By creating a dependency, you’re able to guarantee that a trigger is executed only after the successful execution of a dependent trigger in your data factory.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.