azure data factory pipeline for databricks notebook.

Question

azure data factory pipeline for databricks notebook.

Niren Adhikary 96

Hello,

I am trying to setup a data factory pipeline that will execute multiple data bricks notebooks on ADF in the order like one after another with dependency on the first notebook. At present the notebook runs in parallel. So the idea is if we have multiple notebooks being added it will run in series one after the another.

For better understanding of the pipeline flow. I am sharing below screenshot. I can add another notebook (like notebook 2 in the flow) at the end, basically the user notebooks but when we have multiple notebooks being added then it becomes difficult to keep adding so looking for a way to run them without adding multiple stages or activities.

Kindly suggest.

Thanks,

Niren Adhikary 96 Reputation points

2022-12-20T19:57:45.357+00:00

For better understanding of the pipeline flow. I am sharing the below screenshot.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-01-03T04:51:58.557+00:00

Hi @Niren Adhikary ,

Just checking in to see if the above answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

1 answer

Your answer

Niren Adhikary 96 Reputation points

2022-12-20T19:57:45.357+00:00

For better understanding of the pipeline flow. I am sharing the below screenshot.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-01-03T04:51:58.557+00:00

Hi @Niren Adhikary ,

Just checking in to see if the above answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi @Niren Adhikary ,

Thank you for posting query in Microsoft Q&A Platform.

If I understand correctly, you want to execute multiple notebooks in pipeline. Please correct me if I am wrong.

To execute multiple notebooks, we should consider adding all of them using notebook activity. For example, If we want to run 3 different notebooks, then 3 notebook activities in pipeline.

OR

If all your notebooks are in same workspace, then you can make save all notebook paths in some file or SQL table and read them using lookup activity and then loop that notebooks names using ForEach activity and inside Foreach activity use only one notebook activity. In notebook activity under settings tab use notebook path dynamic expression and pass each notebook name for each execution.

Hope this helps.

-------------

Please consider hitting Accept Answer button. Accepted answers help community as well.

Niren Adhikary 96 Reputation points

2022-12-22T20:31:58.117+00:00

Hi @ShaikMaheer-MSFT
Thanks for your response.

Yes all our notebooks are in same workspace. We have the notebook paths in SQL table. Then we have defined the pipeline flow as shared earlier.

Could you please explain more on below and what we are missing ?
"read them using lookup activity and then loop that notebooks names using ForEach activity and inside Foreach activity use only one notebook activity. In notebook activity under settings tab use notebook path dynamic expression and pass each notebook name for each execution."
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-12-23T05:16:21.78+00:00

Hi @Niren Adhikary ,

Inside foreach activity you need to write expression for file path. Please check below screenshot.

Hope this helps.

------------

Please consider hitting Accept Answer button. Accepted answers help community as well.
Niren Adhikary 96 Reputation points

2022-12-23T15:53:08.043+00:00

Hi @ShaikMaheer-MSFT

We tried that but no luck. The execution runs successfully but cannot run the multiple notebook in the order we wanted to run.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-12-26T05:11:26.707+00:00

Hi @Niren Adhikary ,

From your screenshots, It seems your expression in Notebook path taking some property value from some activity called LKP_USP_GetActivityConfig... Any specific reason why it is like that?

Actually, your notebook is inside ForEach activity, So ForEach activity takes array from lookup activity and iterate over every item from that array. To access each item inside ForEach activity we should use expression as @item(). Let's say your item in array has a property called path then the expression should be as @item().path to access that path property from that array item.

So kindly write your expression accordingly to make sure it runs for every path.

Please check my below video to understand about ForEach activity.
ForEach Activity in Azure Data Factory

Hope this helps.

-----------------

Please consider hitting Accept Answer button. Accepted Answers help community as well.
Niren Adhikary 96 Reputation points

2022-12-27T01:28:02.397+00:00

Our expression seems correct.It works perfectly fine in case of parallel execution.The path is stored in the SQL table which is derived from the look up activity.

Here is an example

https://stackoverflow.com/questions/73844044/data-factory-foreach-activity-run-in-parallel-but-sequentially

Please suggest and workable
solution that should resolve the execution of multiple notebooks in series.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-12-27T06:15:09.68+00:00

Hi @Niren Adhikary , Your expression may validate correctly and give some value. But when you are inside ForEach loop that means, you are taking each item for iteration from the array which you passed to ForEach. And to access that each item you need to use expression as @item() only.

Is this still confusing you? Kindly try as explained in above comment and check to get an idea about what I am trying to explain. If still not clear, then kindly revert back, I will try to have some dummy pipeline and screenshots for you. Thank you.
Niren Adhikary 96 Reputation points

2022-12-27T08:15:38.953+00:00

Hello,

Yes, can you please provide a screenshot of demo pipeline, it will be helpful. Thank you.

We are using the path from the table and first look activity output.I can provide input and output of the pipeline we have setup if that helps you to debug the issue.The notebook path is part of the configuration table so we need to use the look up activity.right ?
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-12-27T08:43:43.887+00:00

Hi @Niren Adhikary ,

In below gif I am using a demoTable which has columns id and notebookPath. Using Lookup activity I am fetching this table data. And in output json of lookup activity I can see both rows came under value property as array of items.

Now, I am passing value property from 'lookupactivity to ForEach to iterate on each item. Inside ForEach activity, In notebook actiivity I am using expression as@item().notebookPath` So that notebook Path property value can be used as path for the notebook.

Hope this helps.

-----------------------

Please consider hitting Accept Answer button. Accepted Answers helps community as well.
26553490 0 Reputation points

2023-11-06T08:57:06.61+00:00

Hi @ShaikMaheer-MSFT

I have same question. 5 notebook. location calling from sql dynamically in notebook activity. But all notebook has different set of parameter. like notebook1 has 2 parameter. notebook2 has 3 parameter and so on. So i cannot base parameter. Because for notebook1 i will create 2 at the time of development then notebook2 will not work. How to handle this parameter passing dynamically

Share via

azure data factory pipeline for databricks notebook.

1 answer

Your answer