Azure Cosmos Data migration from one collection to another

Question

In Cosmos db account -> in a collection(ex: source) I have 7.6 million records, I need to migrate this data to another collection(ex :target and it is a new collection with 0 records) with in a same db account.

I have been told to use .Net core console app to migrate the data from source to target and while transferring data I need to change the Partition key value as well.

when I have executed my script for 30 mins it migrated 36000 records sequentially and I am checking whether the records present in the target collection or not before inserting it.

do we have any better ideas, plans of implementing this and move the data of 7.6 million records into new collection quickly and economical way .. my team dont want to use Azure Data Factory or big data technologies for this.

Is batch migration is better than this ? plz advise

Thanks Vamsi

Answer 1

Vamsi Krishna

Since I haven't heard back from you, I will assume the below.

If you are looking to migrate your data from one collection to another collection in Azure Cosmos DB for either NoSQL API or Mongo API. I would suggest using ADF with copy activity in ADF to import data from one collection to another collection. You need to create a linked service and then follow the steps in the doc.

One option will be to use spark connector Live Migrate Azure Cosmos DB SQL API Containers data with Spark Connector and Azure Databricks, but this is only available for NoSQL API.

Another option using cosmic clone see an example below https://github.com/Microsoft/CosmicClone#Create-backup-copy-of-a-collection

Regards,

Oury