Copy multiple files from multiple folder to single folder in Azure data factory

Question

I have a folder structure like this as a source

Source/2021/01/01/.xlsx files
Source/2021/03/02/.xlsx files
Source/2021/04/03/.*xlsx files
Source/2021/05/04/.*xlsx files

I want to drop all these excel files into a different folder called Output.

Method 1: When I am trying this, I used copy activity and I am able to get Files with folder structure( not a requirement) in Output folder. I used Binary file format.

Method 2: Also, I am able to get files as some random id .xlsx in my output folder. I used Flatten Hierrachy.

My requirement is to get files with the same name as source an dusing above two methods, I am not able to get the desired results

Answer 1

Hello @amikm ,
Thanks for the ask and using Microsoft Q&A platform .
Yes if you do not use the preserve option it copys the the full structure and if you select "preserve" option it gives a the file with guid .

This is what i suggest and I have implemented something in the past and i am pretty confident this should work .

Steps

1. Use getmetada activity and try to loop through all the folder inside Source/2021/
2. Use a FE loop and pass the ItemType as folder ( so that you get folder only and NO files , I know at this time you dont; have file )
3. Inside the IF , add a Execute pipeline activity , they should point to a new pipeline which will take a parameter like
   Source/2021/01/01/
   Source/2021/03/02/
4. The new pipeline should have a getmetadata activity and FE loop and this time we will look for files only .
5. Inside the FE loop add a copy activity and now will have to use the full file name as source name .

Please do let me know how it goes .
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
  • Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of [Q&A Volunteer Moderators][50]