Copy data activity fails when sink/container has immutable storage policy.

Question

I have a source and backup conatiner which are placed in 'mscstatestdev' Azure datalake storage account (HFS enabled). In my Azure data factory pipeline, I'm using a Copy data activity to copy files from source container to backup container before updating the files with latest data.This pipeline runs everyday midnight to copy data in backup conatiner. This normally works, but now I made the backup container immutable by adding a time-based retention policy to it, and the next time the copy data activity ran, it failed with an error (see below). The new files successfully uploaded in backup container with immutable policy when I tried to upload it manually and immutable policy applied on the files.

Is there any way around this, since you should be able to add new files to an immutable container?

Operation on target CopySourcedataFoldertoBackup failed: Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=,Source=,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'mscstatestdev'. FileSystem: 'backup'. Path: 'YY=2023/MM=05/DD=12/abc.parquet'. ErrorCode: 'PathImmutableDueToPolicy'. Message: 'This operation is not permitted as the path is immutable due to a policy.'. RequestId: '7e0e06cb-701f-0017-27b6-8492e8000000'. TimeStamp: 'Fri, 12 May 2023 09:42:28 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'Conflict',Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'mscstatestdev'. FileSystem: 'backup'. Path: 'YY=2023/MM=05/DD=12/abc.parquet'. ErrorCode: 'PathImmutableDueToPolicy'. Message: 'This operation is not permitted as the path is immutable due to a policy.'. RequestId: '65c09dcb-e01f-0067-2eb6-842b1f000000'. TimeStamp: 'Fri, 12 May 2023 09:42:28 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'Conflict',Source=Microsoft.DataTransfer.ClientLibrary,'

Answer 1

Hi @Suchita Phanse ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

I did try to repro the issue and was able to reproduce the error successfully with ADLS Gen2 connector, it failed with above mentioned error and also noticed that an empty file is being created in the container with no data in it. This makes me think that ADF copy process is trying to create an empty file first and the append/add the file data to it which is why it is failing.

This confirms that there is a bug in ADF ADLS Gen2 connector. I'm escalating this issue to respective product teams and will keep you posted as I have more updates from the team.

Workaround: I tried to play around with Regular ADF Blob storage Connector and it seems to work. The files are being created as expected. Hence, to unblock, I recommend using Blob Storage connector as an alternate instead of ADLS Gen2 connector for this requirement until the issue is resolved by product team.

Hope this info helps. Do let me know if you have any questions.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

The immutable storage policy in Azure Data Lake Storage is designed to provide legal hold functionality or to retain data for a certain period of time. Once you enable an immutable policy, the stored data cannot be modified or deleted for the duration specified in the policy.

In your case, it seems that the Copy Data activity in Azure Data Factory is attempting to overwrite or modify existing files in the backup container, which is not permitted due to the immutable policy. The error message 'PathImmutableDueToPolicy' indicates this issue.

However, you mentioned that you're able to manually upload new files to the backup container. The immutable policy should indeed allow you to add new files. It only prevents modifications or deletions to existing files.