ADLS Gen2 backup options

Question

We have set up our data lake using an ADLS Gen2 storage account. Because of using Data Factory and Databricks, we cannot enable the soft delete feature on these accounts. What are our options for backup of the data for accidental deletion? We do have redundancy enabled but that is for DR not the restoration of files.

Thanks!

Answer 1

In addition to enabling soft delete as shared above, you can regularly back up your data to another ADLS Gen2 account in a different region. You can achieve this through:

• ADLCopy: Copy data between ADLS Gen2 accounts.
• Azure PowerShell: Automate data transfer.
• Data Factory: Set up data pipelines for scheduled backups.

Answer 2

Hi @Michael Knutson - Thanks for reaching out.

Are you referring to Databricks Managed Account and facing permissions issue while enabling soft delete? If yes, you can raise a support ticket with the Databricks team to check if there is any feasibility of removing the permissions temporarily to enable those setting and then add them back.

If it is not a Databricks Managed Account, then yes there is no Azure Backup (Operational/Vault) option available. You can then rely on features such as soft delete for blobs/containers in order to protect your data against accidental deletions:
https://learn.microsoft.com/en-us/azure/storage/blobs/soft-delete-blob-overview

https://learn.microsoft.com/en-us/azure/storage/blobs/soft-delete-container-overview

For DR i.e. to have data replicated in another storage account, you might have to copy the data to another account using option such as AzCopy, ADF, Powershell etc. Below are the links to review:

https://learn.microsoft.com/en-us/azure/storage/common/storage-migration-overview?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json#see-also

https://learn.microsoft.com/en-us/azure/storage/common/storage-choose-data-transfer-solution

Please let me know for any further queries and will be glad to assist.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.