Copy new and changed files from FTP server

Question

I am trying to copy new and changed files from an FTP server (incremental loading) , I tried using the solution suggested by Microsoft by adding two parameters : LastModified_From = @Trigger ().outputs.windowStartTime and LastModified_To = @Trigger ().outputs.windowEndTime. The probleme is that in the copy data action I should use the field filter by Last Modified where I call the two parameters , however this field does not exist when the source data is an FTP.

Is there any other solution for incremental loading that could work with an FTP source ?

Thank you

Answer 1

Hi @Anonymous ,

Thank you for posting query in Microsoft Q&A Platform.

Directly incremental load from FTP server using Last modified files files inside copy activity is not possible for FTP source at this moment. Click here to know about available fields for FTP source in copy activity.

You should have logic of filtering your last files first using GetMetaData activity, ForEach activity & IF activity etc. and then perform copy of that files alone.

Below link has similar implementation discussed. Kindly check it.
https://stackoverflow.com/questions/50298122/azure-data-factory-incremental-data-load-from-sftp-to-blob

Hope this will be helpful. Please let us know if any further queries.

--------------------

Please consider hitting Accept Answer button. Accepted Answers helps community as well.