Pick only the latest files everymonth where files are posted on daily basis.

Srikanth Gandam (z) 6 Reputation points
2021-12-07T19:17:51.047+00:00

Hello Forum ,

Can someone help me on the below issue.

I have my source as SFTP where two files are posted on everyday however they are not posting the files on weekend as shown in the below format.
CMF_20211206.txt
DFS_20211206.txt
CMF_20211203.txt
DFS_20211203.txt

My pipeline is scheduled on 5th of every month and it should pick the latest DFS and CMF file available in the SFTP,in the above scenario it should pick the files available on 20211203 .

However I have tried Get metadata Activity with childitems as last Modified date ,due to some issue with the SFTP it is not picking the last modified date.

Can someone help me to resolve this .

Really Appreciate you time and efforts .Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,593 questions
Azure Data Lake Analytics
{count} votes