How to get logs of item deletion past TTL in CosmosDB?

Catherine Du 25 Microsoft Employee

Hello team,

I'd like to know the best way to monitor that the items in the container are getting properly deleted past TTL.

I've come across the Change feed's All versions and deletes mode where I can trigger an Azure functions to continuously pull these changes and check for "timeToLiveExpired" in the response.

This seems to be an overkill for the sole purpose of validating deletion. Besides the All Versions and deletes mode is still in Preview.

Is there any other approach or more straightforward solution that we can validate items are being properly cleaned up?

Best,

Catherine

GeethaThatipatri-MSFT 29,477 Reputation points Microsoft Employee

2024-07-12T12:29:25.6566667+00:00

@Catherine Du Welcome to Microsoft Q&A thanks for posting your question.

could you please elaborate what “properly deleted” means?

Regards

Geetha
Catherine Du 25 Reputation points Microsoft Employee

2024-07-12T18:15:49.4633333+00:00

@GeethaThatipatri-MSFT

Please see below for a detailed version of the questions we had in another post (which was flagged as a duplicate of this one):
Our service currently stores metadata in Cosmos DB with a retention of 24 hours. We have set out TTL to be 18h in the configuration, but we need to track deletion records for compliance reasons.

We came across the Change feed modes in Azure Cosmos DB | Microsoft Learn All Versions and Deletes Mode which contains a "timeToLiveExpired" field. Given the information from your documentation that "Deletes from TTL expirations aren't guaranteed to appear in the feed immediately after the item expires. They appear when the item is purged from the container", how can we track whether the deletion has occurred within retention? Since this mode is on based on a Pull model, is it still possible to get the precise timestamp of when the item is definitely purged?

In the Latest Version mode, you recommend setting a tombstone flag "deleted" with the value true. How does the change feed capture it as an update when the TTL has expired? Does it appear in the Change feed when the item is purged or just upon TTL expiration?
Catherine Du 25 Reputation points Microsoft Employee

2024-07-12T18:19:46.8533333+00:00

@GeethaThatipatri-MSFT
We want to make sure entry is purged or invisible to the users. We are aware that Cosmos DB tombstones the data once it is past TTL, but how can we best prove that?

Share via

How to get logs of item deletion past TTL in CosmosDB?

Your answer