adf/synapse - how to copy all files based on given lastmodified date from source container and save in sink container in respective folder based on extension

Question

adf/synapse - how to copy all files based on given lastmodified date from source container and save in sink container in respective folder based on extension

Wasim 0

Req: Copy files from one container and store in another container- I am specifying the last modified date , i want to copy files only before that last last modified date. The files are of any type. csv,xml,json,txt etc.

Also, in source container my files could be present anywhere , there is no defined hierarchy where the files are stored, some files rightaway in cotainer, some in container/foldera/files, container/folderb/folderb1/files/.

I want to scan all these based on lastmodified condition , and save the files in sink container like -

container/ddata/yyyy/mm/dd/csv/files ,like wise xml,json etc inplace of csv.

Already, i was able to make pipeline that copies files from continer/files. and save into sink container/ddata/yyyy/mm/dd/extension/files.

what i want to do further is- keep the sink structure like- container/somefolder/allfilesinhere.(anyextension,based on lastmodified)

I tried just with copy activity- using container/**/*.csv and using filter last modified option in copyactivity, it copies only files,and is creating the same folder structure as source in the sink.

NIKHILA NETHIKUNTA 4,690 Reputation points Microsoft External Staff

2024-09-19T18:01:28.8033333+00:00

@Wasim
Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Vinodh247 41,971 Reputation points MVP Volunteer Moderator

2024-09-22T10:47:45.5866667+00:00

Wasim: Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members as well as to close this thread out. Community volunteers like us, spend a lot of time in suggesting fix with a proper answer that would assist users. Hence, I would request you to acknowledge the time spent by us.

1 answer

Your answer

NIKHILA NETHIKUNTA 4,690 Reputation points Microsoft External Staff

2024-09-19T18:01:28.8033333+00:00

@Wasim
Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Vinodh247 41,971 Reputation points MVP Volunteer Moderator

2024-09-22T10:47:45.5866667+00:00

Wasim: Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members as well as to close this thread out. Community volunteers like us, spend a lot of time in suggesting fix with a proper answer that would assist users. Hence, I would request you to acknowledge the time spent by us.

Answer 1

Vinodh247 41,971 MVP Volunteer Moderator

Hi Wasim,

Thanks for reaching out to Microsoft Q&A.

Try thte following approach which will allow you to maintain a flat folder structure in the sink container based on file ext. while scanning files from a complex hierarchy in the source container.

Get Metadata Activity:
- Use a Get Metadata activity to recursively list all the files in the source container. In this activity, set the option to list the "Child Items" to retrieve all files, including those in subfolders.
Filter by Last Modified Date:
- In the Get Metadata output, you'll have details of each file. Use a Filter activity to filter files based on the lastModified property. The filter condition can be set to include only those files whose lastModified date is earlier than your specified date.
For Each Activity:
- After filtering the files, use a For Each activity to loop through the filtered files.
Determine File Extension:
- Inside the For Each, use an If Condition or Switch activity to check the file extension (for ex: .csv, .xml, etc.) and create the appropriate path for saving in the sink container based on the file extension.
Copy Activity:
- Use a Copy Activity to copy the files from the source to the sink container. In the destination container path, dynamically build the folder structure as:
```
     
     container/somefolder/allfilesinhere/<extension>
```
File Name Mapping:
- In the copy activity, you can use dynamic content expressions to map the file names appropriately and avoid copying the folder structure. For example, in the sink's file path, you can use expressions to concatenate the year, month, and day based on the file's last modified date, and the file's extension for organizing files.

This approach ensures that the folder structure from the source is not recreated in the sink. Files are saved based on their extension and last modified date in the desired format.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Wasim 0 Reputation points

2024-09-20T06:58:48.8366667+00:00

Hello vinodh, thanks for the reply, inam using get metadata to get the files list, however , after point the dataset to container where all my files and folders and sub folder with files those folders reside, the get metada gives me the output as all file names immediately in the ckntainer, and all foldernames immediately inside the container, it wont go inside those folders and list foles there. In that case how can i recursively retrieve all the files including path until files, coz the path i will need to use in copy activity as well. And get metadata op will inform the copy activity what files to be copied. In the sink i have done the way you hv said it would work but the source? Also other way, when i tried using just copy activity,its source pointed to container and copy recursively turned on and last modified set option set, in the sink of it i have pointed the sink container and folder where I want all these files be copied (without the filename in sink dataset) it copies to sink and creates just the same structure in sink .
Vinodh247 41,971 Reputation points MVP Volunteer Moderator

2024-09-20T07:08:42.1533333+00:00
Use a For Each activity to loop through the output of the Get Metadata activity.

Inside the For Each, use another Get Metadata activity to check if the item is a folder. If it’s a folder, you will again get the child items from this folder and repeat the process recursively until you list all the files.

You can build a custom recursive logic where every time a folder is found, a new Get Metadata is triggered to go deeper inside the folder hierarchy

Once you have all the file paths (with complete folder structure) in an array variable, you can loop through this array in another For Each activity.

Inside this For Each, use a Copy Activity to copy each file to the sink container. In this activity, you can apply the logic for organizing the sink structure based on the file extension
Wasim 0 Reputation points

2024-09-20T07:19:10.5033333+00:00

Thanks, for the second point , getmedata inside foreach, for ex from GM outside foreach ,i am getting folders in container, when passed to GM in for each, it would check the files in those folder yea, but the hierarchy is not defined, like, there could be some files in container/folder/folder/folder/folder/files ,the structure in source is not defined. Is it possible it to go inside these all folders? I mean, wouldn't it need those many GMs as well and when its not defined how deeper the directories could go how to decide upon how many gms to use. Also, can you elaborate upon building custom logic to trigger gm to go inside when folder depth is uncertain. Thanks
Vinodh247 41,971 Reputation points MVP Volunteer Moderator

2024-09-20T08:25:41.4266667+00:00
can you try this?

recursive Logic using ForEach and Variables:

Use a ForEach activity to loop over the output of the Get Metadata and process files/folders.

Use an array variable to keep track of the files discovered.

If a folder is encountered, recursively call another Get Metadata activity to explore the folder and append the files found to the variable.

custom Recursion in ADF using Until Loop:

Use an Until activity to keep checking folders until all files have been discovered. This approach doesn’t require you to know how deep the folder structure is.
NIKHILA NETHIKUNTA 4,690 Reputation points Microsoft External Staff

2024-09-23T16:20:16.61+00:00

Hi Wasim,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

adf/synapse - how to copy all files based on given lastmodified date from source container and save in sink container in respective folder based on extension

1 answer

Your answer