Cosmos DB archiving mechamism

Question

What is the best way to archive records (JSON) older than X months to a cold blob storage storage? Is ADF the answer or there is a less involved/simple way to achieve this?

https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-cosmos-db?tabs=data-factory

Answer 1

Hi @Eliot Chen , welcome to Microsoft Q&A forum.

Azure Data Factory is definitely one of the most commonly used method to migrate/archive data from Azure Cosmos DB. However there are few other ways also to achieve this:

1) Using Azure Cosmos Change Feed: Any new document that gets added to Azure Cosmos db account can be inserted into another storage , however this is not available out of box and we need to write this functionality using azure Function or any other way. This also comes with caveat that change feed does not support 'Delete' currently. This could increase the cost as we are maintaining 2 copies of data simultaneously.

Azure Cosmos DB - Change Feed Lab

2) Azure Cosmos DB Data Migration Tool: Using this we can firstly select the source Azure Cosmos DB account and then write specific query to select the records we want to archive like based on data or any other attribute. After that we can select the target and save the data in JSON format to either local file or Blob.

Source:

Target:

Data migration tool to migrate your data to Azure Cosmos DB

Advantage of using ADF or Azure function is there is no manual intervention here. But if this archival process does not happen too frequently and can be manual then we can choose to use Data Migration Tool as well.

Please let us know if this helps or else we can discuss further on the same. If answer helps, you can mark it 'Accept Answer'.

Answer 2

@AnuragSharma-MSFT, thanks for your feedback!