Latest folder or file extraction (Azure Data factory)

Balasubramanian, N (Nagarajan) 21 Reputation points
2021-06-24T10:55:20.957+00:00

I have 7 years of data stored in separate date folders as partitions or I have 7 years of files in the directory. Requirement is to read the latest file or folder from the directory and then process further. I was able to achieve this using Getmetdataactivity and then a forloop activity for each item in the directory, check the Lastmodified for all 7 years and then read the latest file.

Is there a way to filter the childitems for only last 10 days of modified file or folder? This will avoid looping through the 7 years folder or files.

@Poel van der, RE (Ron)
@MarkKromer-MSFT
@Vishwesh

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,513 questions
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-06-25T06:04:29.757+00:00

    Hi @Balasubramanian, N (Nagarajan) ,

    Welcome to Microsoft Q&A Platform. Thank you for posting your query.

    Please check below detailed explanation.

    In below example, I am trying to Filter Child Items in a folder based on last modified date tie value.
    Step1: My folder has a files which has last modified date times as below

    109252-filesinfolder.png

    Step2: Use get meta data activity and fills values in "Filter by last modified" filed accordingly. In below gif I am trying to filter for last 2 days modified files.
    Expressions Used:

    • Start Time: @adddays(utcnow(),-2,'MM/dd/yyyy HH:mm:ss')
    • End Time: @utcnow()

    109282-getmetadatanew.gif

    Hope this will help. Thank you.

    ---------------------------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a email-notifications.
    0 comments No comments

0 additional answers

Sort by: Most helpful