In Azure Data Factory, what is the process for copying files from a subfolder and appending a the subfolder and timestamp to each file's name in the destination folder?

Question

Hi all,
I have a folder containing several subfolders, each of which contains files. My goal is to create an ADF pipeline that copies all of these files into one folder and appends the subfolder name and a timestamp to each file's name. I was able to retrieve the file names within each subfolder using the getMetadata activity in a for each loop, but I'm unsure how to copy the files and append the subfolder name to their names. Since a nested foreach loop is not possible, and executing another pipeline with the first loop will prevent passing the output of the second getMetadata activity to the pipeline, I'm unsure how to proceed.

Answer 1

Hi moussab orabi,

Thank you for posting query in Microsoft Q&A Platform.

You can achieve it using two pipelines. Pipeline1 to get folder names and iterate over them to get file names and pass them to pipeline2. Pipeline2 iterate over file names and copy them to destination. To generate output file name dynamically, use the expressions accordingly in Sink dataset of copy activity.

Please check below screenshots of my implementation for same.

Step1: Pipeline1, GetMetaData activity to get folder names from root folder. and iterate over them using ForEach activity and pass the file names to Pipeline2 along with folder name.

Step2: Pipeline2, takes filenames along with folder name and iterate over file names and copies it to destination. In Sink dataset use expression to generate file name according to requirement.

Hope this helps.

---

Please consider hitting Accept Answer button. Accepted Answers helps community as well.