Does data factory Copy Activity check if file at source has finished writing ?

Question

Hello,

I am using a Copy Activity to copy recent files (past 15 minutes by checking LastModifiedDate in the copy activity) from SFTP to ADLS Gen 2.

At the moment, to make sure that the files the activity copies are complete in the source and not writing at the moment of copy, I set the LastModifiedDate check to be between T minus 16 mins and T minus 1 mins, T being the pipeline Trigger Time.

My question first is, is this solution necessary ? As in, does the Copy Activity check for complete files before copying or is it even able to copy incomplete files ?
Subsequent question : if a solution like this is necessary, is there any other way to do it ? If a file is being written to my SFTP, will its LastModifiedDate constantly be updated so that a filter on LastModifiedDate < Pipeline Trigger time would be enough ?

Thank you very much !

Answer 1

Hi @Emilien BOUCAUD ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

I have observed a setting "Allow reading files while uploading" in SFTP as below which is by default set to "False". Hence, this would prevent Azure Data Factory to read the file while it is still getting copied.

I don't have SFTP setup so not able to share any repro for the same. And this would cause failure of copy activity but copy activity can still identify those files that are getting copied. To prevent failures, the approach you mentioned - a filter on LastModifiedDate < Pipeline Trigger time should be enough.

Hope this helps! Please let us know if this does not answer the query and we will be glad to assist further.