AI Studio, How can i update data without creating an entirely new vector index?

Question

AI Studio, How can i update data without creating an entirely new vector index?

Anonymous

So I have tried Azure Machine Learning Studio and was exploring AI Studio.

I added some data, created a vector index, created a prompt flow, etc.

Now I have this issue.
I have this vector index that is being used by my prompt flow AI chatbot and i forgot to give it a file while creating it or want to add more.
I see that vector indexes have versions but i see no button or way to create a new version.
User's image

The data used by the index also has versions and does have a button or way to "update" it.
Issue is it seems this undoes all the jobs (chunking etc) that were executed at the initial creation of the vector index.

How can I update or add to the data my vector index uses so that my chatbot can access additional information without having to recreate it every time from scratch and change the reference.

To add to this, how would you do this if i deployed the chatbot to an endpoint. Is it static once deployed?

Thanks in advance.

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-08-19T16:26:18.2633333+00:00

Middendorf, Jelle (ZGNL) I'm looking into this and update you as earliest. Appreciate your time and patience.
Anonymous

2024-09-11T13:24:36.6366667+00:00

@AshokPeddakotla-MSFT Thanks for the help so far. It’s been a few weeks, and I wanted to follow up to express that I’m still very interested in the answer. Could you tell me when I might expect a response? I really appreciate your help.
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-11T16:13:37.11+00:00

Middendorf, Jelle (ZGNL) Apologies for the delayed response. The earlier imaged referenced is in Azure OpenAI studio.

I have an azure blob storage so i should be able to try this out, could you be so kind to explain how you navigated to that window?

Please follow the below steps to get that option in Azure AI studio.

Select the Add data a new data source option in the AI studio -> Select Data Source as Blob Storage where your data is available.

Select next-> choose the search resource you have created or create a new search resource -> you can see the option to select scheduling updates

I hope this clarifies.

And about the deployed chatbots, My chatbot uses RAG so i dont have to train it on the data. So as far as I know the chatbot simply references the data from the vector index. So when the vector index automatically updates with new data added to the blob storage, wouldn't my deployed chatbot now be able to access the updated/new data?

Considering you'd configured RAG, yes the data will be updated as per the schedule.

Do let me know if there are any further queries I can help with.
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-16T14:35:41.47+00:00

Middendorf, Jelle (ZGNL) Did you get a chance to see my suggestions?

Do let me know if you have any further queries.
Lovedeep Singh 230 Reputation points

2024-09-18T10:32:33.8+00:00

Why scheduling option is not available in "Indexes" button under "Components".
Lovedeep Singh 230 Reputation points

2024-09-20T06:47:11.5966667+00:00

After enabling the scheduling of the Index How we can stop this Scheduling? @AshokPeddakotla-MSFT. AshokPeddakotla-MSFT
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-24T03:01:47.6466667+00:00
Lovedeep Singh When you delete a search service, all indexes in the service are deleted permanently.

Currently, there is no option to delete the indexes from Azure AI studio. However, you can follow the below approach.

You can see indexes by going into existing 'search service' resource created in Azure portal and selecting 'indexes' within the search service.

Choose the index you would like to delete and select delete.

Also, you can use Delete Index (Azure Cognitive Search REST API) to delete the indexes.

DELETE https://[service name].search.windows.net/indexes/[index name]?api-version=[api-version] Content-Type: application/json api-key: [admin key]

Do let me know if that helps . If you have any further queries, please create a new issue.

1 answer

Your answer

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-08-19T16:26:18.2633333+00:00

Middendorf, Jelle (ZGNL) I'm looking into this and update you as earliest. Appreciate your time and patience.
Anonymous

2024-09-11T13:24:36.6366667+00:00

@AshokPeddakotla-MSFT Thanks for the help so far. It’s been a few weeks, and I wanted to follow up to express that I’m still very interested in the answer. Could you tell me when I might expect a response? I really appreciate your help.
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-11T16:13:37.11+00:00

Middendorf, Jelle (ZGNL) Apologies for the delayed response. The earlier imaged referenced is in Azure OpenAI studio.

I have an azure blob storage so i should be able to try this out, could you be so kind to explain how you navigated to that window?

Please follow the below steps to get that option in Azure AI studio.

Select the Add data a new data source option in the AI studio -> Select Data Source as Blob Storage where your data is available.

Select next-> choose the search resource you have created or create a new search resource -> you can see the option to select scheduling updates

I hope this clarifies.

And about the deployed chatbots, My chatbot uses RAG so i dont have to train it on the data. So as far as I know the chatbot simply references the data from the vector index. So when the vector index automatically updates with new data added to the blob storage, wouldn't my deployed chatbot now be able to access the updated/new data?

Considering you'd configured RAG, yes the data will be updated as per the schedule.

Do let me know if there are any further queries I can help with.
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-16T14:35:41.47+00:00

Middendorf, Jelle (ZGNL) Did you get a chance to see my suggestions?

Do let me know if you have any further queries.
Lovedeep Singh 230 Reputation points

2024-09-18T10:32:33.8+00:00

Why scheduling option is not available in "Indexes" button under "Components".
Lovedeep Singh 230 Reputation points

2024-09-20T06:47:11.5966667+00:00

After enabling the scheduling of the Index How we can stop this Scheduling? @AshokPeddakotla-MSFT. AshokPeddakotla-MSFT
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-09-24T03:01:47.6466667+00:00

Lovedeep Singh When you delete a search service, all indexes in the service are deleted permanently.

Currently, there is no option to delete the indexes from Azure AI studio. However, you can follow the below approach.

You can see indexes by going into existing 'search service' resource created in Azure portal and selecting 'indexes' within the search service.

Choose the index you would like to delete and select delete.

Also, you can use Delete Index (Azure Cognitive Search REST API) to delete the indexes.

DELETE https://[service name].search.windows.net/indexes/[index name]?api-version=[api-version] Content-Type: application/json api-key: [admin key]

Do let me know if that helps . If you have any further queries, please create a new issue.

Answer 1

AshokPeddakotla-MSFT 35,971 Moderator

Middendorf, Jelle (ZGNL) Greetings & Welcome to Q&A forum!

How can I update or add to the data my vector index uses so that my chatbot can access additional information without having to recreate it every time from scratch and change the reference.

Did you check this build and consume vector indexes in Azure AI Studio already? You can schedule updates. Also, you can choose to update the index hourly or daily.

User's image

Once the Indexer is established, you can also directly modify the schedule settings through the Azure AI Search Portal. See Update or rebuild an index in Azure AI Search and Schedule automatic index refreshes for more details.

To add to this, how would you do this if i deployed the chatbot to an endpoint. Is it static once deployed?

AFAIK, If you want to update the chatbot with new data, you will need to retrain the model and redeploy it to the endpoint.

Do let me know if that helps or have any other queries.

Anonymous

2024-08-14T12:18:03.2266667+00:00

@AshokPeddakotla-MSFT Oh nice, that looks very promising.
So i went through the steps you gave me the link to (https://learn.microsoft.com/en-us/azure/ai-studio/how-to/index-add) and as you can see at step 12 (image below) it shows the option for schedueled updates. But i still have an issue.

I have only worked with uploaded files so far so i thought if i select azure blob storage it must work for me as well. Sadly when I do I dont get shown this option. (Image below)

In your response you show where you can select it but i cant find it. It looks very similar to the image in (https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/use-your-data?tabs=blob-storage%2Ccopilot#schedule-automatic-index-refreshes).
I have an azure blob storage so i should be able to try this out, could you be so kind to explain how you navigated to that window?

And about the deployed chatbots, My chatbot uses RAG so i dont have to train it on the data. So as far as I know the chatbot simply references the data from the vector index. So when the vector index automatically updates with new data added to the blob storage, wouldn't my deployed chatbot now be able to access the updated/new data?

You have been very helpfull so far, thank you very much.

Share via

AI Studio, How can i update data without creating an entirely new vector index?

1 answer

Your answer