Document not deleted when Blob is removed using Indexers Automatically generated from Azure OpenAI Data Studio

Fulwani, Yash 5

I am testing an Azure OpenAI on your data solution. I have set up an Azure Open AI resource and walked through the 'Add your data' workflow with Azure Blob Storage and the backing data source, this generates two indexers in the associated Azure Search resource- one indexer to chunk the data, and one indexer to index the chunks.

Periodically we upload, change, or delete blob in the source container and re-run the indexers. We have observed that when a blob is removed - the associated documents no longer appear in the first index (populated by the indexer that chunks the documents), but does still appear in the final index (populated by the indexer that indexes the chunked documents)- leaving them as orphaned documents.

I am wondering if anyone else is having this issue with orphaned documents when implementing Azure Open AI On Your Data with Azure Blob Storage using the Azure Open AI Data Studio? I am thinking that perhaps: (a) the first indexer is still chunking removed documents, (b) the second indexer is failing to detect or remove deleted documents from where the chunking indexer stores them.

I have confirm that our blob storage containers and the search data sources appropriately meet the requirements outline here that should allow indexers to detect and remove deleted documents.

The default deletion policy that is created in datasource is
"dataDeletionDetectionPolicy": { "@odata.type": "#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy" }

Should I change it?

Thank You

Yash

ajkuma 24,971 Reputation points Microsoft Employee

2024-07-17T10:52:41.9366667+00:00
Thanks for posting this question. I see your comment on similar discussion on this.

I understand you have tried couple things to isolate the issue, while I'm checking on this.
Just to highlight,/ for example, yes, the following policy considers a file to be deleted if it has a metadata property IsDeleted with the value true:

PUT https://[service name].search.windows.net/datasources/file-datasource?api-version=2023-11-01 { "name" : "file-datasource", "type" : "azurefile", "credentials" : { "connectionString" : "<your storage connection string>" }, "container" : { "name" : "my-share", "query" : null }, "dataDeletionDetectionPolicy" : { "@odata.type" :"#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy", "softDeleteColumnName" : "IsDeleted", "softDeleteMarkerValue" : "true" } }

Run the indexer. Once the indexer has processed the file and deleted the document from the search index, you can then delete the physical file in Azure Storage.

Kindly let us know, I'll follow-up with you further.
Fulwani, Yash 5 Reputation points

2024-07-17T13:34:34.0133333+00:00

Hi,

I am not sure what you mean when you say. "Run the indexer. Once the indexer has processed the file and deleted the document from the search index, you can then delete the physical file in Azure Storage." -> I have to run it every time before deleting the file? This doesn't seem correct.

Also the default deletion policy that is created in datasource is
"dataDeletionDetectionPolicy": { "@odata.type": "#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy" }

And there is ".deltrack" file maintained in my blob storage. So it should automatically detect and delete files from the index right?

The problem is "The files get deleted only from 1st index and not the 2nd index. Rather it should get deleted from both the indexes." The 2nd index takes data from chunk datasource and I do not have track of what corresponding chunks of original file are because they are created by skill. So I can only delete original document.

Please look into this issue and suggest what should be done.

Thank You

Yash
ajkuma 24,971 Reputation points Microsoft Employee

2024-07-17T18:57:10.8566667+00:00

Thanks for the follow-up and additional info.

Just to clarify, you mentioned that you created the indexers and configured them. Could you please let us know how exactly did you do it? Was it a manual configuration or using via the portal? Also, if you check the data source settings for "{index-name}-datasource", is the Track deletions box checked or not?
Fulwani, Yash 5 Reputation points

2024-07-17T20:06:08.7466667+00:00

I have not created the indexers. The Azure OpenAI on your data from the portal creates the 2 indexers, 2 indexes, skillset and datasources. When I check the datasource. The Track deletions box is checked and "Data Deletion detection policy says Native blob soft delete" for both the datasources.. original and chunk
ajkuma 24,971 Reputation points Microsoft Employee

2024-07-19T03:33:45.4866667+00:00

Thanks for the follow-up and additional clarity. I'm checking on this and will get back shortly.
ajkuma 24,971 Reputation points Microsoft Employee

2024-07-19T18:49:54.5633333+00:00

Fulwani, Yash, While I'm discussing (and investigating) on this issue internally.

Could you please manually rerun the 2nd indexer after the files were deleted from the chunks container and indicate whether this solves the problem?

If this works, then probably the indexers just ran out of sync. As a workaround for this would be to increase the schedule interval of the 2nd indexer so that the updates to the chunks container are reflected faster. Please let me know your observations.

Thanks for your collaboration and patience!
Fulwani, Yash 5 Reputation points

2024-07-19T19:04:28.5633333+00:00

I tried manually re-running the 2nd indexer. It does not change anything. I am not sure if the files were deleted from chunks container and I am unable to check it, because for a single file in main container there are many chunks in the chunk container. The blob name in chunk container is 64 letter hash which has no relation to its name in original container. So only after opening all files I will be able to check if those files are deleted or not. Also I am not sure how Azure maps files in both containers to delete relevant ones from 2nd container.

I can delete the files from the original container but how do I map and delete relevant ones from chunk container? Do I have to do it manaully everytime?

Thank You

Yash
ajkuma 24,971 Reputation points Microsoft Employee

2024-07-22T19:06:26.0866667+00:00

Apologies for the delayed response from over the weekend. I'm checking this internally and will post back shortly. Thanks!
ajkuma 24,971 Reputation points Microsoft Employee

2024-07-24T18:23:39.3566667+00:00

Yash, I have reached out to privately to fetch additional details about your resources.

Share via

Document not deleted when Blob is removed using Indexers Automatically generated from Azure OpenAI Data Studio