How to Import File in Latest Folder

CB Weiner 65 Reputation points
2023-07-13T01:07:32.06+00:00

The directory looks like: main/project1/. There are daily folders in here, each named by the date i.e. yyyy-mm-dd. There's only 1 file inside each folder but the name changes. The pipeline runs daily. How do you import the file from the latest folder?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,259 questions
0 comments No comments
{count} vote

Accepted answer
  1. ShaikMaheer-MSFT 38,511 Reputation points Microsoft Employee
    2023-07-13T16:16:09.77+00:00

    Hi CB Weiner,

    Thank you for posting query in Microsoft Q&A Platform.

    There are couple of ways to do this.

    Consider using Get Meta Data Activity to get all folder names using child Items field in configuration. Now, take all these folder names into SQL temp table and use SQL engine power (using lookup or script activity) to write query that will give you back latest folder name. Once you get that pass that folder in to Copy activity with parameterized dataset to get file from the folder dynamically.

    OR

    After getting folder names in an array using GetMetaData activity, loop them and find latest one and then use that folder name dynamically in copy activity to copy file. You can check below video for reference to know the process. Get Latest File from Folder and Process it in Azure Data Factory

    Hope this helps. Please let me know how it goes or if any further queries.


    Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.