Duplicate documents inside the index Azure Cognitive Search after Indexer re-run on schedul. Can we overwrite data inside Index?

Question

I created an Azure Cognitive Search using a Cosmos DB SQL API database. I created the index and the indexer. My Cosmos DB database is refreshed daily. Hence, I created the indexer in my azure search to be refreshed on schedule. The problem I have now is my index. The index is duplicated and retrieves duplicate items. When I created my index I had 23,000 documents indexed and currently, I have 46,000 indexed items. My search results are duplicates. This is a production app at work, so your help is much appreciated.

Answer 1

@Redar Ismail Chicho , Thanks for the follow-up and additional info.

You may retrieve the data with Search Documents (Azure Cognitive Search REST API) | Microsoft Learn and get the doc key for the duplicate ones and proceed to Add, Update or Delete Documents (Azure Cognitive Search REST API) | Microsoft Learn as needed using those REST APIs.

I suggest that you review your index schema to make sure you're using an index key that is representative of your unique doc id (such as the database doc ID). If you are getting duplicate items in your index, is because you are not using a matching value of a unique field in the source but perhaps a random id generated by the system and not one defined by you -see (Index overview - Azure Cognitive Search | Microsoft Learn).