Copy csv files from multiple sub folders and sink to adls

Question

My source is adls, having different folders and folders contains csv files.
I need to look in to each folder and copy data dynamically.

storage account having multiple folders.

i used get metadata to get folders and foreach to iterate each folder .
I side foreach i'm executing a pipeline with same getmetedata with brings csv files list and copy data using foreach iteration.

But am unable to pass it dynamically.
Can any one help me with a solution.

If required i can provide folder structure and other details.

Answer 1

Hi @shanucnu8888 , typically you need to have a Lookup activity to lookup through the source files and pass it on to ForEach activity to iterate

through and finally copy activity to copy the data , do not forget to check wildcard option in copy activity. To make it dynamic you can create source data

set parameter and sink data set parameters and in connection tab you can pass this on rather than hardcoding file and folder paths. Thanks

Answer 2

Hello @shanucnu8888 ,

My understanding is that you would want to copy files from your source folders/subfolder to ADLS Folder dynamically. Please correct if I'm not accurate.

If you want to copy a particular set of files, then you will have to pass the file names using lookup activity and pass or your also use get metadata activity to get the list of files and filter them accordingly before processing. This is pretty much similar to what you have done or @Samy-7940s proposed solution.

But in case if you just want to point the source of the Copy activity to a source folder and copy all the csv files from all the subfolders recursively then you can utilize wildcard naming option in copy activity and the copyBehavior settings which would help you copy all files (*.csv) from all the subfolders of a root folder to your desired target ADLS directory.

For more if I recommend going through this article which has more details of the resulting behavior of the copy operation for different combinations of recursive and copyBehavior values. - Copy activity recursive and copyBehavior examples

Hope this will help. Please let us know if any further queries or if your requirement is different than my understanding.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators