Need help in deleting a large number of blob files from blob storage

Question

Hello, we are trying to delete a large number of blob files and are not having much luck. We have our files structured in "folders" using a year/month/day. Our hope was to then easily go in and delete a whole year worth of files, but we have come to learn this is not as easy as it sounds.

Here is what we have found/tried so far...

• It seems if we had used hierarchal storage, that supports actual folder concepts and makes deleting a folder like this much easier. However, when I went to look at the upgrade process to enable this, it says that the upgrade can take several hours and during that time the storage account is offline. So this isn't really an option as a result.
• We tried using the "delete-batch" command via AZ CLI, but after letting this run for 4 hours, it doesn't appear that any blobs had been removed. Example of command we used below:

az storage blob delete-batch --source "ctname" --account-name "stgaccount" --pattern "2020/*"

For reference, when we use Azure Storage Explorer to view statistics on this "2020" folder we are trying to remove, it reports there are 9.2 million blobs. Using the above command works with much smaller containers, but for large ones like this it seems like it never gets anywhere.

Any advice or other ideas for us to try? Thanks!

Answer 1

Hei @jeff tucker ,

Thank you for contacting Microsoft community.

As i understand, You have a year worth files and you want to delete them in bulk.

If i generalize a bit to assume you may have the same problem next year and the year after, so you accumulate files every year and after the year is over you do not need them.

Or simply, all files need to go after they have completed a year.

This can be solved and automated via Azure Storage account lifecycle management policy, these policies enable you to manage the lifecycle of each object in a storage account. It possible to delete or archive among other options based on the number of days since the object's creation or access.

It also allows you to filter blobs to specific prefix etc.

Read more about the lifecycle policies and how to implement them here

If this helped you, please mark it as "answer".

Answer 2

@Jeff Tucker

You can boot a VM in the same region as storage account and utilize scripts to manage this. We have ready-made scripts in azure-cli and PowerShell for this: 

Manage block blobs with Azure CLI - Azure Storage | Microsoft Learn

Manage block blobs with PowerShell - Azure Storage | Microsoft Learn

Let me know the results. Thanks.

Answer 3

Thanks for the suggestions. It did help confirm that there is not an easier way than what I had been trying, so good to know I didn't miss anything. I actually have had some luck with the newest update for Azure Storage Explorer to delete large folders. It takes a long time (several hours over a course of many days) - but it does eventually delete them all. Good to know about the policy option as well, we will consider that to help as well. Thanks!