Latest file extraction (Azure Data factory)

Question

Hi Team,

I want to extract latest file from Azure data lake store based on last modified date and want to process in another folder.Could you please someone let me know what needs to be done here if anyone has idea or implemented the same.

for example i do have 3 files

"lastModified": "2020-10-22T08:21:53Z",
"fileName": "people.csv",

"lastModified": "2020-10-22T07:51:42Z",
"fileName": "Product.csv",

"lastModified": "2020-10-22T14:48:51Z",
"fileName": "Address.csv",

So it should process only "Address.csv" file.

I am looking forward your response.

Thanks,
Pankaj

Answer 1

Hello @pankaj chaturvedi
Thanks for the ask and also using the Microsoft Q&A.

We have created the below pipeline and our test shows that its should do the trick .

Logic :
1 . We will use 2 of the Getmetadata activity one for iterating the folder which have the files and the other to get the metadata for the specific file
( in this case the lastmodified date). The seond is a paramterized one and will pass the file name to that .

2.We are using an if clause to check the the lastmodified date , I am using the tick function ( as it return an int , if you wish we can use anything else )

@greater(ticks(activity('get file details').output.lastModified),ticks(formatDateTime(variables('TakeAnyStartDate'))))

3.All the variables are self explanatory , but wanted to call out the TakeAnyStartDate , iam using this to set the any value to start with .
I am using this as value "2000-10-27"

Thanks Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 2

I have explained this in a detailed video.
please watch.
https://youtu.be/9YU0RiGLijY

Answer 3

You may take below for reference and test to see what works better -

https://stackoverflow.com/questions/60558731/get-the-latest-added-file-in-a-folder-azure-data-factory
https://stackoverflow.com/questions/57374617/copy-new-and-changed-files-by-lastmodifieddate-with-azure-data-factory-v2
https://stackoverflow.com/questions/64374678/only-copying-the-latest-folder-files-for-a-specific-day-with-azure-data-factory

---

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav