Latest folder or file extraction (Azure Data factory)

Question

I have 7 years of data stored in separate date folders as partitions or I have 7 years of files in the directory. Requirement is to read the latest file or folder from the directory and then process further. I was able to achieve this using Getmetdataactivity and then a forloop activity for each item in the directory, check the Lastmodified for all 7 years and then read the latest file.

Is there a way to filter the childitems for only last 10 days of modified file or folder? This will avoid looping through the 7 years folder or files.

@Poel van der, RE (Ron)
@MarkKromer-MSFT
@Vishwesh

Answer 1

Hi @Balasubramanian, N (Nagarajan) ,

Welcome to Microsoft Q&A Platform. Thank you for posting your query.

Please check below detailed explanation.

In below example, I am trying to Filter Child Items in a folder based on last modified date tie value.
Step1: My folder has a files which has last modified date times as below

Step2: Use get meta data activity and fills values in "Filter by last modified" filed accordingly. In below gif I am trying to filter for last 2 days modified files.
Expressions Used:

• Start Time: @adddays(utcnow(),-2,'MM/dd/yyyy HH:mm:ss')
• End Time: @utcnow()

Hope this will help. Thank you.

---------------------------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a email-notifications.