Chunk index AI Studio BYOD

Matis Kofoed Petersen 20 Reputation points
2024-09-11T10:26:50.9366667+00:00

It seems that creation of Indexes in AI Studio changed in August, when you create a Index now its created with 2 Skillsets Microsoft.Skills.Text.SplitSkill and Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill.

This follows the documentation https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization#using-integrated-vectorization-during-indexing, but no index or indexer is created for the chunks

"An indexer, which retrieves raw data from a supported data source and serves as the pipeline engine.
A vector index to receive the chunked and vectorized content."

If I look at older index created prior to any updates in AI Studio, we see that its made with Microsoft.Skills.Custom.WebApiSkill, and in this custom skillset parameters for the Chunk index is presence and the Chunk index and indexer is created so 2 index and indexers is created.

I did a comment on it here: https://learn.microsoft.com/en-us/answers/questions/2032550/automatic-indexer-creation-is-not-working-in-azure

So how do I get the chunking index and indexer back, without using the "old" custom skillset.

Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
3,104 questions
{count} votes

Accepted answer
  1. santoshkc 8,950 Reputation points Microsoft Vendor
    2024-09-12T14:35:50.6966667+00:00

    Hi @Matis Kofoed Petersen,

    Thank you for your patience.

    Azure AI Search has undergone updates that now include integrated data chunking and vectorization directly within the indexing process using standard skillsets like Microsoft.Skills.Text.SplitSkill and Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill. This shift may have replaced the need for custom skillsets like Microsoft.Skills.Custom.WebApiSkill that previously handled chunking. I tried reproducing the issue by adding new data, and I was able to retrieve the information successfully, indicating that the system is functioning as intended.

    If the new setup doesn't meet your specific needs, you can consider customizing the process by manually configuring chunking through a custom skillset. This approach should help you achieve similar functionality to what was available before the update.

    I hope this helps! Thank you.

    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.