ADF Delete Activity not deleting Folders

Question

I have a pipeline to delete some folders from ADLS. My folder structure is as below.

/raw/MainFolder/SubfolderA/20230430/File.parquet
/raw/MainFolder/SubfolderA/20230415/File.parquet
/raw/MainFolder/SubfolderA/20230410/File.parquet

/raw/MainFolder/SubfolderB/20230430/File.parquet
/raw/MainFolder/SubfolderB/20230420/File.parquet
/raw/MainFolder/SubfolderB/20230405/File.parquet

The pipeline is currently deleting the File.parquet under the date folder. But I need it to delete the whole folder named with date (20230430).

Currently the pipeline is deleting the parquet files when they exist and erroring out when the folder is empty. I am not passing the parquet file name to my pipeline. (Just the folder path) I have recursively enabled in the delete activity.

Error: Failed to execute delete activity with data source 'AzureBlobStorage' and error 'The required Blob is missing. Folder path: raw/MainFolder/SubfolderA/20230430/.'.

How do I get the delete activity to delete the folder itself and not just the files in it?

I am very new to Azure data factory. Appreciate your help.

Pipeline is deleting the files but not the folders

Answer 1

Hi Vinay Gundamaneni,

If your data set specifically resolves to any file, it wont delete the folder.

I just tried to replicate, but it is deleting, all files inside a specific folder, along with the folder.

Request you to refer to this URL where you have all the use cases.

Hope this helps, if not please provide more details, like data set settings\ delete activity settings for deeper investigation.

Awaiting your response,

Suba