Copy Data setting "Delete files after completion" leads to errors

Question

So we've been using the copy data activity for some time now, and since last week it starts failing.

We're daily copying about 10 million files from blob storage to ADLSGen2.

The failure is a bit odd, it says "user error []".

While doing some testing on development it turns out that when we disable the option "Delete files after completion" on the source, the copy activity doesn't fail. Nothing has changed from our side. It was running fine for at least a month.

To me it seems like the delete files after completion might be removing some files that are processed duplicate due to parralel processing, because, the file mentioned in the error did end up in the sink.

Like said, the problems started last week, en we didn't do any release. Also, there's no logic to when it stops, sometimes it processes 100.000 files and fails, but most of the time it stops after 1500 files or something like that.

I tried playing around with parallelism and fault tolerance, but nothing helps. The only thing that helps is turning of delete files after completion, but that is not an option in production.

{
    "errorCode": "2200",
    "message": "ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'xxxx.json'.,Source=Microsoft.DataTransfer.ClientLibrary,'",
    "failureType": "UserError",
    "target": "copy from landing zone",
    "details": []
}

Answer 1

Hi @RickMsBi ,

Thank for using Microsoft Q&A forum and posting this question. As you mentioned that it worked for a month without issues and started failing since a week ago, seems like something from product side has to be investigated to understand the actual root cause of the problem. For deeper investigation if you have a support plan, would recommend support ticket for deeper investigation. In case if you don't have a support plan, please let me know and I can work with you offline in creating a support ticket.

Mitigation: To mitigate the problem, you may workaround by having your copy activity to copy files from a particular date and time range using file last modified date field under source settings in your copy activity by utilizing dynamic expression to define particular date range in combination withwild card file path. And then have a subsequent Delete activity to delete the files once after they have been copied. You can utilize combination of wild card file path and Filter by last modified option in both Copy activity and Delete activity to copy and delete same set of file range. This way you can disable Delete files after copy option until it is investigated, and the root cause is fixed.

Copy activity settings:

Delete Activity settings:

Please let us know if you have any questions regarding the workaround implementation or in opening a support ticket to further investigate on the initial issue with out of box feature (Delete files after copying).

Thank you.

Answer 2

Hi @RickMsBi ,

Thank you for the ask. We experienced the same issue while we were trying to move files from a SFTP server. File size was big in our case. The workaround for us was to unchecked the option from Copy Activity and add a separate DELETE activity with the same file mask. Thanks!