How to efficiently migrate MongoDb to azure CosmosDB with azure Databricks?

Question

While searching for a service to migrate our on-premise MongoDB to Azure CosmosDB with Mongo API, We came across the service Azure Data Bricks. We have total of 186GB of data. which we need to migrate to CosmosDB with less downtime as possible. How can we improve the data transfer rate for that. If someone can give some insights to this bigdata related spark based PaaS provided by Azure, It will be very much helpful. Thank you

Answer 1

Hi @Jithin M V ,

Thanks for using Microsoft Q&A !!
Your performance to migrate data will depend on few factors and can be modified using the below configurations -

1. Number of workers and cores in the spark cluster
2. maxBatchSize
3. MongoDB Spark practitioner and partition key

So, in order to increase the data transfer rate you need to adjust number of workers to execute tasks and maxBatchSize will help you control the rate at which data will be saved to Azure Cosmos DB. Also, you need to disable indexes during data transfer to improve your data transfer rate more.

Please refer to the Optimize the migration performance for details.
Please let me know if you have any additional questions.

Thanks
Saurabh