Azure Search bulk document deletion via REST API not deleting everything requested.

Question

I am using the REST API to perform a batch deletion of multiple documents. I have no indexer, nor do I want one. When I issue a batched deletion of say 300 Document Ids, my request is accepted and I am returned a list of the 300 Document Ids and their deletion status. Each returned Document Id is marked as successful with a 200 HTTP response on each record.

However, subsequent queries against the index (even after letting a half-hour expire), I am still left with a random set of documents still in the index (close to half). My last run had about 308 document Ids. After issuing the batch delete, I was left with about 130 document Ids that were reported to be deleted. I then issued a batch delete for the 130 left over docs, and am still retrieving 1 document id left in the index.

This does not seem to be an "eventual consistency" issue. I could understand a few minutes of wait time before the index was fully updated, but after waiting a half hour and still seeing data there that should not be, it seems like an issue with the service. Is this something that is being addressed, or am I required to perform this task in a loop until I get the desired (correct) state of the index?

Answer 1

Hi @Behr, Todd
It seems you are having issues with bulk deletion of documents in Azure Search. Although the REST API shows successful deletion requests (with a 200 HTTP response), the documents may still remain in the index

• Deletions might not be immediately reflected in the index due to the background process that updates the index statistics.
• Deleting a non-existent document will still return a 200 status code, which can make it seem like the document was deleted.
• Concurrent operations modifying the same documents can cause version conflicts, preventing some deletions.
• Heavy service load may cause some delete requests to not be processed as expected. Implementing a retry mechanism or throttling requests might help.

check the status of the documents still in the index to ensure they are not being recreated or modified by other processes. A loop to check the state of the index after deletions might be necessary.

References:

• Add, Update or Delete Documents (Azure AI Search REST API)
• Bulk delete in Azure API for FHIR
  Let me know if you have any further assistance needed.