Latest file extraction by scanning multiple folders and copy to azure sql database

Anmol Ganju 166 Reputation points
2022-01-14T09:29:29.293+00:00

Hi, I have a folder structure in ADLS Gen2 in container source that goes as below:

/table1/2021/01/20/file1.parquet
/table1/2021/02/20/file2.parquet
/table1/2022/01/10/file3.parquet
/table1/2022/01/12/file4.parquet
/table2/2021/01/12/file5.parquet
.
.
.
.

Like wise I have multiple tables..

I want to copy the content of latest file as per current datetime to the Azure SQL Database, How this can be achived, we are looking for only copying the latest file to azuresqldatabase.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,609 questions
0 comments No comments
{count} votes

Accepted answer
  1. Samy Abdul 3,366 Reputation points
    2022-01-14T10:50:43.58+00:00

1 additional answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2022-01-14T15:56:52.797+00:00

    Hi @Anmol Ganju ,

    Thank you for posting query in Microsoft Q&A Platform.

    You can use GetMetaData activity to get Last Modified Date Time vale of file. So you should consider using it for all your files and then try to compare each other then finally filter our latest file to copy.

    Similar kind of logic implemented in below video. Kindly check that to get an idea of doing it.
    https://www.youtube.com/watch?v=QHeG36oqkPQ

    Hope this will help. Please let me know if any concerns.

    ---------------------

    Please consider hitting Accept Answer button. Accepted answers helps community as well.