Total data stored in Datalake Gen2 contains soft delete?

Question

I would like to know if the metric provided by Azure Monitor for total data stored contains soft deleted data or if it is not taken into account in this measurement?

Thank you very much

Answer 1

@Luis Ladron de Guevara Moreno Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

The metric provided by Azure Monitor for total data stored does not include soft deleted data. This means that when you check the total data stored in Azure Data Lake Storage Gen2, the soft deleted data is not taken into account in this measurement.

Additional information:

The metric provided by Azure Monitor for total data stored in Azure Storage accounts includes soft deleted data. Soft deleted data is data that has been marked for deletion but is still recoverable within the retention period.

The metric for total data stored in Azure Storage accounts is calculated based on the size of the blobs, files, tables, and queues stored in the account, including any soft deleted data. This means that if you have soft deleted data in your storage account, it will be included in the total data stored metric.

If you want to exclude soft deleted data from the total data stored metric, you can use the List Blobs API to retrieve the size of the blobs in your storage account and filter out any blobs that have been soft deleted. You can then calculate the total size of the remaining blobs to get the total data stored metric without soft deleted data.

I hope this helps! Let me know if you have any further questions.

Blob soft delete

When enabled for a storage account, blob soft delete allows you to specify a retention period for deleted objects between 1 and 365 days. This period indicates how long the data remains available after it's deleted or overwritten. After the retention period has expired, the blob is permanently deleted. You can enable blob soft delete for an existing storage account using the Azure portal. 

• Container soft delete Container soft delete is a data protection feature that works similarly to blob soft delete, but for containers. You can specify a retention period for deleted containers between 1 and 365 days, with a default of seven days. During the retention period, you can restore a soft-deleted container and its contents to the state it was in when it was deleted. You can enable or disable container soft delete for a storage account using the Azure portal, PowerShell, Azure CLI, or an Azure Resource Manager template. 
• Azure Backup soft delete Azure Backup soft delete is an opt-in feature that's enabled by default on all newly created vaults. You can extend the retention duration for deleted backup data from 14 to 180 days. The first 14 days of retention are free, but you'll be charged for any period beyond that. 

Soft delete for containers

When calculating data capacity in Azure Blob Storage with soft delete enabled, it's important to note that soft deleted data is still counted towards the total storage capacity until it is permanently deleted. This means that even though the data is marked as deleted, it still occupies space and is billed at the same rate as active data until the retention period expires

If you have both soft delete and versioning enabled, when you delete the current version of a blob, a version of the blob is created which is not soft deleted and contributes towards storage capacity

If you need to monitor and calculate the storage capacity, you can use tools like Azure Storage Explorer or run PowerShell scripts to get detailed insights into your storage usage

References

Soft delete for blobs capability for Azure Data Lake Storage is now generally available | Azure updates | Microsoft Azure

Please let us know if you have any further queries. I’m happy to assist you further.    

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.