Bulk deletion of blobs using Lifecycle Management rule

Question

Bulk deletion of blobs using Lifecycle Management rule

Saksham Chhabra (SACHHABRA) 0

Hi Azure team,

I want to delete 2 yrs of blob data using lifecycle management and henceforth delete/archive data every 3-7 days, want to check since data is huge could delete operation in bulk using lifecycle management with rules cause any impact on writes for new slices. Is it the recommended approach for historical data cleanup in blobs.

Anand Prakash Yadav 7,855 Reputation points Microsoft External Staff

2023-11-23T12:59:29.3233333+00:00

Saksham Chhabra (SACHHABRA): Just checking in to see if the below-answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
Saksham Chhabra 10 Reputation points Microsoft Employee

2023-11-23T13:59:28.27+00:00

Yes that helps. Just wondering if historical data is pretty huge (200TBs), can it impact parallel writes happening in other containers of the same storage account since time for deletion might be high. The blobs which needs to be delete follow this sort of structure (folder), container -> blob -> sub blobs -> parquet files. For deletion, all the parquet files belonging to multiple sub blob folders need to be deleted. So the rule should have prefix added (sub blob) path
Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-11-23T17:16:32.94+00:00

Saksham Chhabra: No, it should not impact parallel writes. The lifecycle rule would be running without hindering the writes from any other services in your adls storage. Since the size is huge, I would suggest start slow to find out how it fares for maybe 10TB first and then gradually increase.
Saksham Chhabra 10 Reputation points Microsoft Employee

2023-11-24T05:25:02.76+00:00

Thanks. One last question on a production account, is configuring the policy via azure portal the right way to proceed then.
Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-11-24T07:04:07.32+00:00

You can do it on any method like portal, powershell, cli etc., but portal would be easier for the first time. You can explore each points for this time and can setup a code that can be reused from next time.
Anand Prakash Yadav 7,855 Reputation points Microsoft External Staff

2023-11-28T12:23:18.01+00:00

Saksham Chhabra (SACHHABRA): Just checking in to see if the below-answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

1 answer

Your answer

Anand Prakash Yadav 7,855 Reputation points Microsoft External Staff

2023-11-23T12:59:29.3233333+00:00

Saksham Chhabra (SACHHABRA): Just checking in to see if the below-answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
Saksham Chhabra 10 Reputation points Microsoft Employee

2023-11-23T13:59:28.27+00:00

Yes that helps. Just wondering if historical data is pretty huge (200TBs), can it impact parallel writes happening in other containers of the same storage account since time for deletion might be high. The blobs which needs to be delete follow this sort of structure (folder), container -> blob -> sub blobs -> parquet files. For deletion, all the parquet files belonging to multiple sub blob folders need to be deleted. So the rule should have prefix added (sub blob) path
Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-11-23T17:16:32.94+00:00

Saksham Chhabra: No, it should not impact parallel writes. The lifecycle rule would be running without hindering the writes from any other services in your adls storage. Since the size is huge, I would suggest start slow to find out how it fares for maybe 10TB first and then gradually increase.
Saksham Chhabra 10 Reputation points Microsoft Employee

2023-11-24T05:25:02.76+00:00

Thanks. One last question on a production account, is configuring the policy via azure portal the right way to proceed then.
Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-11-24T07:04:07.32+00:00

You can do it on any method like portal, powershell, cli etc., but portal would be easier for the first time. You can explore each points for this time and can setup a code that can be reused from next time.
Anand Prakash Yadav 7,855 Reputation points Microsoft External Staff

2023-11-28T12:23:18.01+00:00

Saksham Chhabra (SACHHABRA): Just checking in to see if the below-answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Answer 1

Vinodh247 34,661 MVP Volunteer Moderator

Hi Saksham Chhabra (SACHHABRA):

Thanks for reaching out to Microsoft Q&A.

Yes, it is a recommended approach and there won't be any impact on your new data stores. You can find the steps here...

https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview

https://stackoverflow.com/questions/74555693/how-to-delete-blobs-from-blob-storage-using-lifecycle-management

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Saksham Chhabra (SACHHABRA) 0 Reputation points

2023-11-30T07:49:11.02+00:00

Yes, this is helpful. Just one last question, there is a soft delete policy active on storage account, the lifecycle management rule deleted the blobs, I cross checked in the container, although the storage of account in metrics view on portal still seems to be about the same. Is it due to soft delete ?
Anand Prakash Yadav 7,855 Reputation points Microsoft External Staff

2023-12-05T11:43:41.22+00:00

Yes, the behavior you're observing is consistent with the use of Azure Storage soft delete. When soft delete is enabled for a storage account, deleted blobs are retained for a specific retention period before being permanently removed. During this retention period, the storage metrics might not immediately reflect the reclaimed storage space.

If you need to immediately reclaim storage space after blob deletions, you may consider adjusting the soft delete retention period or manually purging the soft-deleted blobs.
Anand Prakash Yadav 7,855 Reputation points Microsoft External Staff

2023-12-06T12:11:21.9433333+00:00

Saksham Chhabra (SACHHABRA): Just checking in to see if the below-answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
Anand Prakash Yadav 7,855 Reputation points Microsoft External Staff

2023-12-08T10:29:37.9966667+00:00

Saksham Chhabra (SACHHABRA): Just checking in to see if the below-answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Share via

Bulk deletion of blobs using Lifecycle Management rule

1 answer

Your answer