How to apply retention policy on specific data in azure data explorer

Question

Hi there,
I need to remove specific data from azure data explorer automatically after some conditions are met, but currently what I understood from retention policy document we only can specify a retention policy on the whole table or database but not each row.

To provide more details, I have an asp.net core application which ingests data continuously to adx. Each data item should be removed automatically based on their type and retention days specified for that type.

So I'm wondering if this is possible and if not would you please recommend a solution for this?

Many thanks

Answer 1

Hello @Mahsa Mohammadi ,

Azure Data Explorer uses an insert-only way of storing data by default.

This means records cannot be changed and, more crucial, deletion of data is handled differently from eg. relational databases like SQL Server.

By default, records have a certain TTL, based on the cluster, database, or table-level settings.

This does not mean that individual records cannot be removed. This is needed in some cases like GDPR regulation.

Please check out the purge command documentation for Azure Data Explorer.

Also, check the notes and warnings:

...It is not designed to support frequent delete requests, or deletion of massive quantities of data, and may have a significant performance impact on the service.

Notice this is not a fully automated process.

You need to build your own process (like an Azure Function or the power platform support (eg. Logic Apps)) that executes the Purge command.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.