File Transfer and Deletion Workflow

Question

I am in the process of transferring files from Azure File Storage to Blob storage (Archive tier) using Azure Data Factory (ADF). In addition to the transfer, it is crucial for me to delete the files from the source once the copying process is complete. It is essential to maintain the same file hierarchy in the source, which includes multiple folders, at the destination.

While the ADF copy activity successfully manages the file transfer, I am encountering a challenge regarding the deletion of files from the source after the copy operation. Is there a specific option within the ADF copy activity that allows me to delete only the source files that have been successfully copied?

If the copy activity is not the ideal solution for this task, are there alternative options I should consider? I appreciate your guidance.

Additionally, I have a requirement to copy only those files that have not been updated in the last 60 days. How can I implement a filter to selectively copy files based on this criterion?

Answer 1

Hi @Clover J ,
Thank you for reaching out to us with your query.
As I understand you want to delete the files from source once copy is done by using ADF copy activity. If this is not, please let me know.

• Copy activity does not have the ability to delete files after copying. It is data flow which has the option to delete source file or move it a folder upon successful copy. But copy activity is cheaper than data flow. What you can do now is, you can use another copy activity to copy the files to archive and then a delete activity to delete the file after 2nd copy (to archive) is successful.
• You can make use of a get meta data activity and use the last modified date option there to pick the files which were last modified 60 days ago. Please find below screenshot, where in my case, i am filtering files older than 10 days. I have four files in Azfs container, out of which just two files were last modified 10 days ago.

Expression used

Copy

@addDays(formatDateTime(utcNow(),'MM/dd/yyyy'),-10) : Start time UTC
@formatDateTime(utcNow(),'MM/dd/yyyy') : End Time UTC

Set variable is optional, if you want to see how the expression works, you may assign the above expression and see how it brings date.

My output after execution

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.