Recursive delete activity in Azure data factory

Question

I'm going through ADF and was exploring about delete activity. There if we pass folder and set resurively then it will delete everything inside the folder. Now I thought to do some experiment and build logic where I have some folder names inside an array and a starting relative path for the folders. Let's say the relative path is adlscontsiner/so/data and we have folder array with folders A, B inside data. Now I created a flow where I am adding folder name after the path and getting the childitems using metadata activity and then deleting files inside it. Lately I thought what if we get folder in childitems For example in A we have C, D folder So I should iterate them also in the same way. I created a logic where I was calling the above pipeline in a until variable but it was hard to maintain the relative path scope. Suggest me an iterative approach logic for a custom solution to delete files and folders recursively

Answer 1

You can define a pipeline parameter basePath to pass the initial relative path and variables to hold intermediate values such as currentPath, childItems, and foldersToProcess.

Your pipeline structure should be the following :

• Metadata Activity: To get the child items of the current path.
• ForEach Activity: To iterate over each child item.
• If Condition Activity: To check if the item is a file or folder.
• Delete Activity: To delete files.
• Append Variable Activity: To keep track of folders for further processing.
• Until Activity: To iteratively process each folder.

Steps to follow :

Create a pipeline with a parameter basePath (for example adlscontainer/so/data).

Get Child Items:

• Add a Metadata Activity to get child items of basePath.
  • Settings: Set the path to @pipeline().parameters.basePath.
  • Field list: Select childItems.

ForEach Activity:

• Add a ForEach Activity to iterate over childItems from the Metadata Activity.
• Inside the ForEach, add an If Condition Activity to differentiate between files and folders.

If Condition Activity:

• Condition: @equals(item().type, 'Folder')
• If true (it's a folder):
  • Use Append Variable Activity to add the folder path to foldersToProcess.
• If false (it's a file):
  • Use Delete Activity to delete the file.

Append Variable Activity (for folders):

• Variable: foldersToProcess
• Value: @concat(pipeline().parameters.basePath, '/', item().name)

Until Activity (Iterative Folder Processing):

• Add an Until Activity to process each folder in foldersToProcess recursively.
• Condition: @equals(length(variables('foldersToProcess')), 0)
• Inside Until:
  • Add a Get Metadata Activity to get child items of the first folder in foldersToProcess.
  • Use the same ForEach and If Condition logic to handle the new child items.
  • Remove the processed folder from foldersToProcess using a Set Variable Activity.