Azure Data Factory Get First Last Modified File

Question

Hi,

I have a GetMetadata Activity returning Arguments: Child Items and Last Modified,
is there an easy way to add a filter step which just says give me the first Modified Date File?

e.g. if GetMetadata step returns three files:

file1.xml - modified date 17-JAN-21
file2.xml - modified date 01-JAN-21
file3.xml - modified date 24-JAN-21

I only want the filter to return:

file2.xml - modified date 01-JAN-21

Thanks.

Answer 1

Hi @Jay ,

Thanks for your response. In order to copy the First Last modified file, you can follow the same steps described in this thread: ADF: copy last modified blob
But in the If condition activity, instead of @greater() function you will have to use @less() function since you want to copy the first Last modified in other terms first modified file.

And also for the pipeline reference date variable you will have to assign the pipeline trigger time varReferenceDateTime = @pipeline().TriggerTime instead of varReferenceDateTime = 1900-01-01 00:00:00.

This implementation should help you achieve your requirement.

Do let us know how if it goes

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Corey Blimey that's a lot of steps to get the first or last file :-)
Thanks KranthiPakala I'll take a look.

Failing this I guess my next option would be to use filename which contains the file created date and time:

2021-04-08_05-03-16.xml
2021-04-08_03-01-10.xml

I.e. strip this out and write to CosmosDB and then Select out with an Order By FileCreatedDateTime desc TOP 1 then use this as a variable to select the file,
none of it sounds straight forward.

Would be nice to have in ADF a Metadata Sort Task which you could place after current Get Metadata to be able to select a Get Metadata field and Sort asc or desc and return Top x i.e. Top 1 or 10 etc.

Thanks,
J