How to do file deletion tracking for Azure AI Search and Azure File Storage

Question

I have an Azure AI Search resource as well as an Azure Storage resource that has files uploaded to a File Share

I've managed to connect an Index to the File Share, but deletion tracking seems to be a major pain point. I have to use the SoftDeleteColumnDeletionDetectionPolicy as a deletion policy, but there I do not know which column or metadata aspect to track.
I saw the documentation here https://learn.microsoft.com/en-us/azure/search/search-howto-index-changed-deleted-blobs?tabs=portal#soft-delete-strategy-using-custom-metadata

But it seems to suggest I add a new metadata field to every file I have (~1,800 files) and then when I want to delete a file or files, I must first edit that metadata field, which doesn't sound like a scalable or serious method for a deletion of 10-100 files at a time

Is there documentation for which property of a file can change upon deletion? or some other way of achieving deletion tracking for a File Share

Answer 1

Hello Support Boris Shabash,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you are having issues with file deletion tracking for Azure AI Search and Azure File Storage.

Solution

To answer your questions.

Is there documentation for which property of a file can change upon deletion

When it comes to tracking file deletions in Azure File Shares, there is not specific documentation on which property of a file changes upon deletion. But you can infer deletions through other properties and mechanisms such as:

• Last Modified Timestamp.
• File Metadata.
• Activity Log and Azure Monitor.
• Using soft delete for Azure AI Search
• In Azure Storage, add a custom metadata key-value pair to the file to indicate it’s flagged for deletion. You can create a property like "IsDeleted" and set it to true when you want to delete the file.

https://stackoverflow.com/questions/57500954/automatically-delete-files-in-storage.

https://techcommunity.microsoft.com/t5/itops-talk-blog/azure-monitor-detecting-and-alerting-on-file-changes/ba-p/286327.

https://learn.microsoft.com/en-us/azure/architecture/ai-ml/architecture/search-blob-metadata.

some other way of achieving deletion tracking for a File Share

Managing deletion tracking for Azure File Share is essential, especially when dealing with a large number of files.

• You can find out who deleted a file share from your Azure storage account by checking the Activity log in the Azure portal
• Azure Files indexer (preview) in Azure AI Search is one of the best.
• To set up alerts specifically for file deletions, you can create an alert rule using Azure Monitor.
• Create a time-triggered Azure Function app and use an Azure CLI script to periodically delete files that haven't been modified within a specified time frame using Schedule Function App.

https://stackoverflow.com/questions/72710134/how-to-set-alert-on-edit-delete-of-a-file-or-folder-in-azure-portal.

https://stackoverflow.com/questions/68500637/how-to-delete-from-files-from-azure-file-share-periodically-after-say-x-days.

https://learn.microsoft.com/en-us/azure/storage/files/storage-files-monitoring.

References

Use above links as the source to read more and for detail steps.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

Answer 2

Hello Support Boris Shabash,

Greetings! Welcome to Microsoft Q&A Platform.

I understand that you would like to find a deletion tracking methods for a File Share and AI search.

Adding to above information, Azure Files offers soft delete for SMB file shares. Soft delete allows you to recover your file share when it is mistakenly deleted by an application or other storage account user. Refer to this article which provides detailed information on Prevent accidental deletion of Azure file shares

Accidental delete protection for Azure file shares using Azure Backup : To provide protection against cyberattacks or accidental deletion, Azure Backup has added one more level of security to the Azure file shares snapshot management solution by providing protection against the accidental or malicious deletion of backed-up file shares. Now, even if a malicious actor deletes the file share, the file share’s contents and recovery points (snapshots) are retained for a configurable retention period, allowing the successful and complete recovery of source contents and snapshots with no data loss.

Is there a way to trigger an alert when a folder is deleted? Are you referring Azure File share file/folder? Or for complete Storage account? For Azure Fileshare we don't have that option for now! However, you can monitor Azure files Read/Write/Delete/ Azure File share is Alerts : Monitoring Azure Files

How to create alerts for Azure Files

Additional information: You can set alert for Blob storage(container) deletion and Alert when Storage account is deleted.

You can also check via Activity logs - you can find out who deleted the file share folder from the Azure storage account by checking the Activity log in the Azure portal. Navigate to your storage account, and then view the activity log to identify the user responsible for the deletion, refer article.

Hope this answer helps! Please let us know if you have any further queries. I’m happy to assist you further.

---

Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.