How can I append the name of all files in Folders and Sub-Folders inside the same Variable in Azure Data factory?

Anonymous

I have a requirement in Azure Data Factory where there are files in Folders and Sub-Folders. I have highlighted the structure of the same below for reference.

I want to Understand how I can use an Array Variable and append all the filenames inside the variable and set the array as the variable's Value.

ShaikMaheer-MSFT 37,816 Reputation points Microsoft Employee

2022-05-19T15:52:19.817+00:00

Hi @Anonymous ,

Just checking in to see if the below answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

1 answer

Nandan Hegde 29,886 Reputation points MVP

2022-05-04T04:12:55.893+00:00

Hey,
You can use Get metedata activity to get the list of all files within a folder.
In case of sub folders, reiterate over the new path.
Then iterate it over via For activity and use Append variable activity to append the file names in an Array variable
Please sign in to rate this answer.
Anonymous

2022-05-04T04:36:02.36+00:00

Hi,

Thanks for your Answer but it does not work actually. I have already tried to use the getMetadata Activity. The issue is getMetadata Activity will only retrieve the files and Folders which are direct children of the current Folder or Container mentioned in the Dataset of getMetadata Activity and does not traverse recursively and fetch all files at 2nd ,3rd and 4th Levels.

For Instance, In the above mentioned case getMetadata Activity will only retrieve Files1-4.csv and Sub-folder 1&2. I want to traverse down Sub-folders 1&2 and get the files present in there too.

Thanks again for your efforts to answer the question. Let me know if you have a working solution

Nandan Hegde 29,886 Reputation points MVP

2022-05-04T04:43:13.933+00:00

Hey,as stated above , we iterate recursively .
below are the links :
https://richardswinbank.net/adf/get_metadata_recursively_in_azure_data_factory
https://stackoverflow.com/questions/64504159/get-all-files-names-in-subfolders-azure-data-factory
Sign in to comment