How can we run the different level of folders in a heirarchy in batches?

Mansi Yadav 40 Reputation points
2024-05-28T16:27:31.67+00:00

I have a doubt if we can approach something like this in ADF. I have created a for each activity where I'm iterating through an array of folders. Let's say I'm passing folder I'd of A and then I'm calling a pipeline in for each that will return sub folders and then I'm passing that subfolder array to another pipeline which has until loop and iterates through all nested folders. I did this to run the folders in batches but it was still taking lot of time.

So, what I did is I have created a pipeline where I'm passing array of folder ids getting there sub folder ids array and then passing the array to another pipeline that has foreach and gets the folders inside the subfolder and then pass it to the next pipeline that is until loop. This approach helped me to reduce the time by half. So we are running folders in batches and inside those folder we are running subfolders in batches.

In above image this is for initial level I. e. It will iterate folders in batches. Instead of passing subfolders to an pipeline that has unitl loop and logic to handle nested folders, I will pass subfolder array to another pipeline that has foreach activity and we will get subfolders inside them and will send that to the nested logic pipeline.

Now I'm thinking to have a component where we can define a count that will follow above steps. Let's say if I have set count to 3 now we will iterate through the 3level of folder structure and in above approach we are running 2levels in batches and two run 1 more level we need to add a for each. Is this possible in ADF?

I want to automate this. I tried to do this but failed and I Conculded that we need to manually add one more foreach activity for

running three levels in batches.

If you have any solution than let me know.

thumbnail_image002

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
{count} votes