Why does my search index keep referencing older documents no longer in data?

Question

I am working on Azure AI Studio with a deployed model gpt 4o to query documents. I have a data store connection where my documents lie/ index and a job in azure machine learning to reindex the connected datastore everyday. I remove old files from the connected datastore and expect the job to get only new data while i ask questions. The new data is a part of answer but i still get the older answers as well which gives wrong answers. I do not want the index to have older data. It is even removed from the datastore. How do i solve this?

Answer 1

Harsh Bangad Greetings!

As you have mentioned you already removed the old data, one reason could be that the indexer is not detecting the deleted files and is still keeping them in the index.

What kind of data store are you referring to?

I understand that you have already checked that the data is removed. But sometimes the data might not be deleted completely.

Make sure that the old files are removed from the connected datastore completely. You can do this by checking the datastore directly to confirm that the files are no longer present.

Incase if you are referring to Azure Storage, see Change and delete detection using indexers for Azure Storage in Azure AI Search

Also, see Update or rebuild an index in Azure AI Search and let me know if that helps.