How to read latest files from sub folders in Azure Data Factory

Question

I have a s3 bucket with many country folders and each country folder have a zipped csv file.

country_a / report / extracts_2022-Oct-01.zip ( 2022-Oct-01.csv )
country_a / report / extracts_2022-Oct-02.zip ( 2022-Oct-02.csv )
country_b / report / extracts_2022-Oct-01.zip ( 2022-Oct-01.csv )
country_b / report / extracts_2022-Oct-02.zip ( 2022-Oct-02.csv )
country_c / report / extracts_2022-Oct-01.zip ( 2022-Oct-01.csv )
country_d / report / extracts_2022-Oct-01.zip ( 2022-Oct-01.csv )

Using ADF I need to read latest extracted csv file in each country folder :

country_a -> 2022-Oct-02.csv
country_b -> 2022-Oct-02.csv
country_c -> 2022-Oct-01.csv
country_d -> 2022-Oct-01.csv

Is it possible only use 1 pipeline to read latest extracted csv file in each country folder and copy latest csv data to azure sql?

Thank you.

Answer 1

Hi @Galih ,

Thank you for posting query in Microsoft Q&A Platform.

Kindly check below video where I explained about get latest file folder and process it in azure data factory. In the video, I used azure storage account, in your case its S3 bucket. So connector which we use while creating dataset alone will change logic of implementation will be same. So kindly check it and implement the same.
Get Latest File from Folder and Process it in Azure Data Factory

Hope this helps. Please let me know if any further queries.

---------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification