How to add a vectorizer for a search index in C#

ThibaultVerlinde-4578 120 Reputation points
2024-03-07T14:26:30.9433333+00:00

Hiya

I'm currently working on a project that indexes a certain amount of data. This goes flawlessly when I create the index, with vectors, inside the portal.
Whenever I create such index with a C# program, it succeeds, but without the vectorizer. There isn't also any info on how/where you add a Vectorizer inside the new index with that code.

Thanks in advance!

EDIT: I found the answer myself, but for some reason i can't accept my own answer. See the answer posted by me to find the solution.

My code:

//Initialisation of new index
var indexName = _srchClient.IndexName;
var vectorSearchConfigName = "vectorconfig-" + _options.ProjectName;
var vectorSearchProfileName = "vectorprofile-" + _options.ProjectName;
//var semanticConfigName = "semanticconfig-" + _options.ProjectName;
var index = new SearchIndex(indexName)
{
    VectorSearch = new()
    {
        Algorithms =
        {
            new HnswAlgorithmConfiguration(vectorSearchConfigName)
        },
        Profiles =
        {
            new VectorSearchProfile(vectorSearchProfileName, vectorSearchConfigName)
        }
    },
    Fields =
    {
        new SimpleField("id", SearchFieldDataType.String) { IsKey = true },
        new SearchableField("content") { AnalyzerName = LexicalAnalyzerName.StandardLucene },
        new SimpleField("sourcePage", SearchFieldDataType.String) {IsFacetable = true},
        new SimpleField("sourceFile", SearchFieldDataType.String) {IsFacetable = true},
        new SearchField("embedding", SearchFieldDataType.Collection(SearchFieldDataType.Single))
        {
            VectorSearchDimensions = 1536,
            IsSearchable = true,
            VectorSearchProfileName = vectorSearchProfileName
        }
    },
    //When using free tier, this can't be used!
    //SemanticSearch = new()
    //{
    //    Configurations =
    //    {
    //        new SemanticConfiguration(semanticConfigName, new()
    //        {
    //            ContentFields =
    //            {
    //                new SemanticField("content")
    //            }
    //        })
    //    }
    //}
};
// Create a new search index
await _srchIndexClient.CreateIndexAsync(index);   
Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
684 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,159 questions
0 comments No comments
{count} votes

Accepted answer
  1. Azar 18,045 Reputation points
    2024-03-11T11:00:35.82+00:00

    Hi again Thibault Verlinde

    Thats awesome finding the solution, so let me paste your solution below as you cant accept your own answer, accept the answer and this helps the other community members who face the same problem. Thanks very much for the contrubution.

    This is the updated code:

    //Initialisation of new index
    var indexName = _srchClient.IndexName;
    var vectorSearchConfigName = "vectorconfig-" + _options.ProjectName;
    var vectorSearchProfileName = "vectorprofile-" + _options.ProjectName;
    var vectorizerName = "vectorizer-" + _options.ProjectName;
    //var semanticConfigName = "semanticconfig-" + _options.ProjectName;
    var index = new SearchIndex(indexName)
    {
        VectorSearch = new()
        {
            Algorithms =
            {
                new HnswAlgorithmConfiguration(vectorSearchConfigName)
            },
            Profiles =
            {
                new VectorSearchProfile(vectorSearchProfileName, vectorSearchConfigName)
                {
                    Vectorizer = vectorizerName
                }
            },
            Vectorizers =
            {
                new AzureOpenAIVectorizer(vectorizerName)
                {
                    AzureOpenAIParameters = new AzureOpenAIParameters()
                    {
                        ResourceUri = new Uri(_options.OAIEndpoint),
                        ApiKey = _options.OAIKey,
                        DeploymentId = _embeddingConfigs.EmbeddingModel
                    }
                }
            }
        },
        Fields =
        {
            new SimpleField("id", SearchFieldDataType.String) { IsKey = true },
            new SearchableField("content") { AnalyzerName = LexicalAnalyzerName.StandardLucene },
            new SimpleField("sourcePage", SearchFieldDataType.String) {IsFacetable = true},
            new SimpleField("sourceFile", SearchFieldDataType.String) {IsFacetable = true},
            new SearchField("embedding", SearchFieldDataType.Collection(SearchFieldDataType.Single))
            {
                VectorSearchDimensions = _embeddingConfigs.VectorSearchDimensions,
                IsSearchable = true,
                VectorSearchProfileName = vectorSearchProfileName
            }
        },
        //When using free tier, this can't be used!
        //SemanticSearch = new()
        //{
        //    Configurations =
        //    {
        //        new SemanticConfiguration(semanticConfigName, new()
        //        {
        //            ContentFields =
        //            {
        //                new SemanticField("content")
        //            }
        //        })
        //    }
        //}
    };
    
    1 person found this answer helpful.
    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Azar 18,045 Reputation points
    2024-03-07T14:56:38.41+00:00

    Hey there Thibault Verlinde

    Thats a good question thanks for using QandA platform

    So i guess you need to define and configure the vectorizer within the SearchIndex object before creating the index.

    Lemme modify your code to include a vectorizer, below

    var indexName = _srchClient.IndexName;
    var vectorSearchConfigName = "vectorconfig-" + _options.ProjectName;
    var vectorSearchProfileName = "vectorprofile-" + _options.ProjectName;
    
    var vectorizer = new SearchIndexVectorizer
    {
        Name = vectorSearchConfigName,
        Source = "/document/content", // Specify the source field for vectorization
        Tokenizer = new LexicalTokenizer(LexicalTokenizerName.StandardLucene),
        OdataType = "#Microsoft.Azure.Search.StandardAnalyzer"
    };
    var index = new SearchIndex(indexName)
    {
        VectorSearch = new()
        {
            Algorithms =
            {
                new HnswAlgorithmConfiguration(vectorSearchConfigName)
            },
            Profiles =
            {
                new VectorSearchProfile(vectorSearchProfileName, vectorSearchConfigName)
            }
        },
        Fields =
        {
            new SimpleField("id", SearchFieldDataType.String) { IsKey = true },
            new SearchableField("content") { AnalyzerName = LexicalAnalyzerName.StandardLucene },
            new SimpleField("sourcePage", SearchFieldDataType.String) {IsFacetable = true},
            new SimpleField("sourceFile", SearchFieldDataType.String) {IsFacetable = true},
            new SearchField("embedding", SearchFieldDataType.Collection(SearchFieldDataType.Single))
            {
                VectorSearchDimensions = 1536,
                IsSearchable = true,
                VectorSearchProfileName = vectorSearchProfileName
            }
        },
        Vectorizers = { vectorizer } // Add the vectorizer to the index
    };
    // Now ccreate a new search index
    await _srchIndexClient.CreateIndexAsync(index);
    
    

    If this helps kindly accept the answer thanks much.