Immutable Storage with Spark Delta Table

Question

Immutable Storage with Spark Delta Table

Abhiram Duvvuru 236 Microsoft Employee

Hi Team - We want to process data from source system on daily cadence using spark and store the data as delta table in ADLS Gen2 in immutable state . Is there way to add constraint/enforce delta table to always append data from source i.e If uses/application tries to delete the data from delete table , it should throw error. Please let me know to how to achieve it.

In addition, I have looked into storage immutability at the container level and enabled legal hold but delta write in not working on subsequent write ( i.e Step1: read data from source for Day 1 and write to delta table Step2: Enable Immutability at container level Step3: read from source for Day2 and append to delta table (below error)

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-01-10T22:06:43.353+00:00

Hello @Abhiram Duvvuru ,

Welcome to the MS Q&A platform.

Regarding your first question, I will check with my internal team and get back to you.

Regarding the error: "This operation is not permitted as the path is immutable due to one or more legal holds"

This means the data in the path you are trying to write is placed under legal hold. So it cannot be modified or deleted. This feature allows to protection of certain data from accidental deletion or modification by establishing a legal hold on it.

If you want to change or delete the data, you'll need to remove the legal hold on the container or the directory.

Reference document: https://learn.microsoft.com/en-us/azure/storage/blobs/immutable-legal-hold-overview
Abhiram Duvvuru 236 Reputation points Microsoft Employee

2023-01-10T22:55:28.22+00:00

Thank you @Bhargava-MSFT ! I have looked into it. I see if we remove legal hold/time-policy then anyone can delete the container. Our Upstream data source supports immutability and when we process data in spark (synapse notebook) we want to support immutability as well. Please let me know if there is any alternative solution. (Note: we always want to append the data from source on daily cadence)

Thanks, Abhiram

1 answer

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-01-10T22:06:43.353+00:00

Hello @Abhiram Duvvuru ,

Welcome to the MS Q&A platform.

Regarding your first question, I will check with my internal team and get back to you.

Regarding the error: "This operation is not permitted as the path is immutable due to one or more legal holds"

This means the data in the path you are trying to write is placed under legal hold. So it cannot be modified or deleted. This feature allows to protection of certain data from accidental deletion or modification by establishing a legal hold on it.

If you want to change or delete the data, you'll need to remove the legal hold on the container or the directory.

Reference document: https://learn.microsoft.com/en-us/azure/storage/blobs/immutable-legal-hold-overview
Abhiram Duvvuru 236 Reputation points Microsoft Employee

2023-01-10T22:55:28.22+00:00

Thank you @Bhargava-MSFT ! I have looked into it. I see if we remove legal hold/time-policy then anyone can delete the container. Our Upstream data source supports immutability and when we process data in spark (synapse notebook) we want to support immutability as well. Please let me know if there is any alternative solution. (Note: we always want to append the data from source on daily cadence)

Thanks, Abhiram

Answer 1

The expected behavior with immutable blobs, we should be able to create new files on the immutable blobs, but we can't append/modify the data on the existing files.

Current behavior: Immutable works only when we upload the files manually, but we cannot create any new files on the Immutable blobs when using the synapse notebook. This is with delta as well as for parquet writes.

Error message: we will see the below error when using synapse notebook

"This operation is not permitted as the path is immutable due to one or more legal holds"

The product team is currently looking into the issue. I will update this thread as soon as I hear from them.

Share via

Immutable Storage with Spark Delta Table

1 answer

Your answer