Azure OpenAI: using functions and bring your own data in parallel doesn't work

GuenterR 5

We have an Azure OpenAI instance with uploaded data (bring your own data). Using the OpenAIClient from azure-sdk-for-js it is possible to chat with the instance and get answers composed from the uploaded data. This is described in https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/openai/openai/samples/v1-beta/javascript/bringYourOwnData.js and it works as expected. In addition we want to retrieve dynamic data from a database via function calls. This is described in https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/openai/openai/samples/v1-beta/javascript/functions.js it also works well. When I add the parameters for both in getChatCompletions only function calls are working. Access to uploaded data is not possible. API Versions: 2023-12-01-preview, 2023-08-01-preview. I checked the behavior with direct http-Rest requests to OpenAI using Postman and got the same result. BYOD and functions work when called in different requests but not together in one request.

Ramr-msft 17,616 Reputation points

2024-01-30T15:25:44.46+00:00

Thanks for the question, However, it’s important to note that the functions and function_call parameters have been deprecated with the release of the 2023-12-01-preview version of the API. The replacement for functions is the tools parameter, and the replacement for function_call is the tool_choice parameter. As for the Bring Your Own Data (BYOD) feature, it allows you to run supported chat models such as GPT-35-Turbo and GPT-4 on your data without needing to train or fine-tune models. The model has access to, and can reference specific sources to support its responses. If you’re still facing issues, it might be helpful to capture the network request sent to your AOAI /extensions/chat/completions API and then map the parameters to your code to call SDK. Also, ensure that you have the necessary permissions to perform the operation.
GuenterR 5 Reputation points

2024-01-31T10:37:14.6666667+00:00
Thanks for your answer. Using the JS SDK parameters functions and functionCalls (JS format) still works well. I tried tools and toolChoice (!) instead and got an error. I tested with

openAIClient = new OpenAIClient( "https://openai-healthcare-dom.openai.azure.com/", new AzureKeyCredential("44adb0be593c4ac58b326ca81ea9b404") , { apiVersion: "2023-12-01-preview" } );

GuenterR 5

First testcase with cognitive search and without functions:

    response  = await openAIClient.getChatCompletions("gpt-4", conv, {   
      azureExtensionOptions: {
        extensions: [
          {
            type: "AzureCognitiveSearch",
            endpoint: openAiSearchEndpoint,
            key: openAiSearchKey,
            indexName: openAiSearchIndex
          },
        ],
      }, 
    });

Question for information in our uploaded documents are answered correctly.

GuenterR 5

Second testcase with functions:

    response  = await openAIClient.getChatCompletions("gpt-4", conv, {  
          functionCall: "auto", functions: functions 
    });

My functions works well!

GuenterR 5

Third testcase with functions and with cognitive search:

    response  = await openAIClient.getChatCompletions("gpt-4", conv, {  functionCall: "auto", functions: functions , 
      azureExtensionOptions: {
        extensions: [
          {
            type: "AzureCognitiveSearch",
            endpoint: openAiSearchEndpoint,
            key: openAiSearchKey,
            indexName: openAiSearchIndex
          },
        ],
      }, 
    });

My functions still work but questions correctly answered in testcase 1 got an unspecific answer

GuenterR 5 Reputation points

2024-01-31T10:50:10.2466667+00:00

We need both dynamic information collected by functions and static information from uploaded documents.
Luis Angel Maya Rueda 0 Reputation points

2024-04-25T00:42:39.2366667+00:00
Hello everyone,

I have a question related to the topic "Bring Your Own Data". I am using an Azure OpenAI resource and importing my data from Azure AI Search. However, when I click on "View Code" and select "CURL", I notice some parameters that are not included in the GitHub Code https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/openai/openai/samples/v1-beta/javascript/bringYourOwnData.js.

My question is: Can I set up these parameters in the code? For example, can I add semantic search as shown in the following JSON? I would like to set up everything in fact.

12 { "dataSources": [ 13 { 14 "type": "AzureCognitiveSearch", 15 "parameters": { 16 "endpoint": "'$search_endpoint'", 17 "indexName": "'$search_index'", 18 "semanticConfiguration": "vector-semantic-configuration", 19 "queryType": "vectorSemanticHybrid", 20 "fieldsMapping": {}, 21 "inScope": true, 22 "roleInformation": "You are an AI assistant that helps people find information.", 23 "filter": null, 24 "strictness": 3, 25 "topNDocuments": 5, 26 "key": "'$search_key'", 27 "embeddingDeploymentName": "text-embedding-ada-002" 28 } 29 } 30 ]

Luis Angel Maya Rueda 0

Hello everyone,

I have a question related to the topic "Bring Your Own Data". I am using an Azure OpenAI resource and importing my data from Azure AI Search. However, when I click on "View Code" and select "CURL", I notice some parameters that are not included in the GitHub Code https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/openai/openai/samples/v1-beta/javascript/bringYourOwnData.js.

My question is: Can I set up these parameters in the code? For example, can I add semantic search as shown in the following JSON? I would like to set up everything in fact.

{
12  "dataSources": [
13    {
14      "type": "AzureCognitiveSearch",
15      "parameters": {
16        "endpoint": "'$search_endpoint'",
17        "indexName": "'$search_index'",
18        "semanticConfiguration": "vector-1713196904123-semantic-configuration",
19        "queryType": "vectorSemanticHybrid",
20        "fieldsMapping": {},
21        "inScope": true,
22        "roleInformation": "You are an AI assistant that helps people find information.",
23        "filter": null,
24        "strictness": 3,
25        "topNDocuments": 5,
26        "key": "'$search_key'",
27        "embeddingDeploymentName": "text-embedding-ada-002"
28      }
29    }
30  ]

GuenterR 5 Reputation points

2024-04-25T05:11:10.5466667+00:00

Hi Luis,

The link you posted is the description of the SDK. CURL output shows the HTTP request. SDK generates the HTTP request from the input parameters which looks similar to the parameters in the request but they are different. So it depends whether you want to use the SDK or setup the HTTP requests by your own.

Regards

Guenter

2 answers

SchildmannDominik-3228 5 Reputation points

2024-02-07T15:05:55.78+00:00

NOTE: The concurrent use of Chat Functions and Azure Chat Extensions on a single request is not yet supported. Supplying both will result in the Chat Functions information being ignored and the operation behaving as if only the Azure Chat Extensions were provided. To address this limitation, consider separating the evaluation of Chat Functions and Azure Chat Extensions across multiple requests in your solution design.

From https://learn.microsoft.com/en-us/dotnet/api/overview/azure/ai.openai-readme?view=azure-dotnet-preview Assuming it's similar in the JavaScript SDK, although I'm not sure why it's the other way around for you.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment
GuenterR 5 Reputation points

2024-02-08T06:54:42.58+00:00

Hi Dominik, Thanks for your answer. This note is missing on the Javascript SDK site. I already tried to separate functions and cognitive search in different requests but this didn't work well.
Today I've changed the option parameters from "functions" (still works) to "tools". Now cognitive search is performed and functions in tools are ignored when both are sent in options.
Please sign in to rate this answer.
Andrew Lloyd Bulacan 0 Reputation points

2024-02-29T08:29:49.9766667+00:00

Do you manage to make it work together?

GuenterR 5 Reputation points

2024-02-29T09:07:39.62+00:00

Hi Andrew, we're currently working on a design to let a rule-based bot decide whether interface to Azure Open AI is called with parameters "function" or "cognitive search". As far as I know both functionality in one call is possible with Open AI and I hope this will be available soon with Azure Open AI.

Andrew Lloyd Bulacan 0 Reputation points

2024-03-04T01:01:08.2566667+00:00

Hi GuenterR, thank you for your response. We might check that as well. Cheers!
Sign in to comment