Doesn't the index of AzureCognitiveSearchChatExtensionConfiguration operate as the index defined in the search service?

Question

Hello.

I'm developing an OpenAI service that can provide answers based on Delimited type csv files in Blob storage using Cognitive search.

In the Search service in Azure Portal, I was able to define the index to search with the type I wanted.

When I created OpenAIClient using Dotnet and entered same index into the IndexName of Extensions, I was able to check that the index had a different format when I checked the citations.

This is the search result in Search service.

"@search.score": 1,
"id": "2",
"first_name": "Florencia",
"department": "Marketing",
"AzureSearch_DocumentKey": 	"aHR5jc3Y7Mg2",
"metadata_storage_name": "DUMMY_DATA.csv"

and, This is the result cited by OpenAIClient in dotnet.

"content": "Research and Development\nKane\n5",
"id": null, 
"title": null, 
"filepath": null,    
"url": null, 
"metadata": {
        "chunking": "orignal document size=8. Scores=4.343202Org Highlight count=3."
},      
"chunk_id": "0"

I'm curious why it works this way. Additionally, if other settings are needed in Extensions, can you tell me how to set the key in the desired way for a Delimited type file whose first line includes a header?

Answer 1

@김상권 The index defined in the Azure Cognitive Search service and the index used in the AzureCognitiveSearchChatExtensionConfiguration are two different indexes. The index defined in the Azure Cognitive Search service is used to store and search the data, while the index used in the AzureCognitiveSearchChatExtensionConfiguration is used to map the data from the search index to the chat extension.

When you create an AzureCognitiveSearchChatExtensionConfiguration, you need to specify the index name, the field mappings, and other settings. The field mappings define how the data from the search index is mapped to the chat extension. If the field mappings are not defined correctly, you may get unexpected results.

In your case, it seems like the field mappings are not defined correctly, which is why you are getting different results in the search service and the OpenAIClient. You need to define the field mappings correctly to map the data from the search index to the chat extension.

To set the key in the desired way for a Delimited type of file whose first line includes a header, you can use the "delimitedTextHeaders" parameter in the indexer definition. This parameter specifies the headers in the delimited text file. For example:

{
  "name": "my-indexer",
  "dataSourceName": "my-data-source",
  "targetIndexName": "my-index",
  "parameters": {
    "configuration": {
      "parsingMode": "delimitedText",
      "delimitedTextHeaders": "id, name, department"
    }
  }
}

Here, "id", "name", and "department" are the headers in the delimited text file. You can replace them with the headers in your file.