How to pull data from cosmos db (mongo db) to azure data explorer using data factory without duplicates

Question

Hey,

We have a requirement where we need to replicate Azure cosmos DB for Mongo DB collection to a Azure data explorer table.

• Using copy data tool we were able to copy this data from our mongo db collection to our data explorer table. But each time the pipeline runs and pulls data all records will be created again in data explorer.
• We tried also building a dataflow to perform this task, but cosmos DB for Mongo DB is not supported as a source currently.

How can we pull data from mongo db collection our azure data explorer table (with a certain frequency) without ending up with duplicate records?

Answer 1

Welcome to the MS Q&A platform.

To avoid duplicate records, one option is by using control command in Azure data explorer command activity and clear the Kusto table before loading data into the Kusto table using copy activity.

Management (control commands) overview

Video tutorial for Azure data explorer activity

clear table data command

I hope this helps.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.