SFTP Function to move files to archive folder and delete the files from the current location

Question

SFTP Function to move files to archive folder and delete the files from the current location

steve hawk 6

Hi All,I have a data pipeline to get the files from an SFTP Server and COPY into ADLS Gen2. Now i want to add an additional functionality. After the COPY Activity i want to move those files to an Archive folder and delete the files from the Current folder. I have done this using SSIS in the past , but not with ADF. Please any advice or document will be great.
Thanks

Vaibhav Chaudhari 38,916 Reputation points Volunteer Moderator

2020-10-01T14:53:45.763+00:00

Please try adding extra Copy activity, configure source and target folders and tick the below option which deletes files from source after copy is completed

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav
steve hawk 6 Reputation points

2020-10-01T15:30:12.567+00:00

Thanks Vaibhav will update you.
steve hawk 6 Reputation points

2020-10-01T18:33:58.007+00:00

Vaibhav,
I do not see the option to check delete files

1 answer

Your answer

Vaibhav Chaudhari 38,916 Reputation points Volunteer Moderator

2020-10-01T14:53:45.763+00:00

Please try adding extra Copy activity, configure source and target folders and tick the below option which deletes files from source after copy is completed

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav
steve hawk 6 Reputation points

2020-10-01T15:30:12.567+00:00

Thanks Vaibhav will update you.
steve hawk 6 Reputation points

2020-10-01T18:33:58.007+00:00

Vaibhav,
I do not see the option to check delete files

Answer 1

KranthiPakala-MSFT 46,642 Microsoft Employee Moderator

Hi @steve hawk , @Vaibhav Chaudhari , as per my analysis deleterFileAfterCompletion property is only available when you use binary format dataset.

If your dataset format is other than binary, then in order to delete the files after copying, chain your Copy activity to a Delete activity to delete the files from source location.

To explore about Delete activity, please refer to this doc: Delete Activity in Azure Data Factory

Below data stores are supported for Delete activity:

Azure Blob storage
Azure Data Lake Storage Gen1
Azure Data Lake Storage Gen2
Azure File Storage
File System
FTP
SFTP
Amazon S3
Google Cloud Storage
HDFS

Hope this helps. Let us know if you have further query.
Thank you

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2020-10-05T20:01:28.27+00:00

Hi @steve hawk ,

Following up to see if the above information was helpful. If you have further query, do let us know.

Thank you

----------

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2020-10-06T19:27:39.377+00:00

Hi @steve hawk ,

We still have not heard back from you. Just wanted to check if the above suggestion was helpful? In case If you already found a different solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue. Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Thank you

Share via

SFTP Function to move files to archive folder and delete the files from the current location

1 answer

Your answer