How to auto-update vector search index

Question

I am using Azure AI Studio to create a chatbot with some custom data, but I am experiencing several problems. The data is contained a BLOB container, and using the API in .NET via an indexer, every time I add/remove files from the BLOB container I perform the indexing operation to update the index, so far no problem. The problem arises when I use vector research to search for data, in fact, even adding/removing files in the BLOB container and re-executing the indexing operation the "old" data that was uploaded in the "upload data" section of Azure AI Studio remains present. My question is why when I upload the data source from Azure AI Studio, not selecting the vector research and update the files in the BLOB container and re-execute the indexing then I can chat with the new data, but when I do the same thing but selecting the vector research I can't chat with the new updated files as well (even though the app is being deployed)

Answer 1

Hello @Francesco Granozio

are you using the same type of data when using vector search? what is the size of your data set

In meantime, here are a few things to consider:

1. There might be errors during the indexing process. Azure AI Search stops indexing when the error count exceeds ‘maxFailedItems’. If you want indexers to ignore these errors (and skip over “failed documents”), consider updating the maxFailedItems and maxFailedItemsPerBatch
2. Azure AI Search imposes limits on vector storage. If vector size exceeds this limit, any further indexing requests result in failure. You can resume indexing once you free up available quota by either deleting some vector documents or by scaling up in partitions
3. Some older search services created before January 1, 2019 are deployed on infrastructure that doesn’t support vector workloads. If you try to add a vector field to a schema and get an error, it’s a result of outdated services. In this situation, you must create a new search service to try out the vector feature
4. Data Formats and File Types: Azure OpenAI on your data supports several file types including .txt, .md, .html, Microsoft Word files, Microsoft PowerPoint files, and PDF4. Make sure your data is in a supported format.

Please let us know if you have further questions

Best,

Grace