Azure OpenAI 'Bring Your Own Data' Feature Not Providing Correct Answers

Kaja Sherief 95

I've been using Azure's OpenAI and the 'Bring Your Own Data' (BYOD) feature for a while now. Recently, I've encountered an issue where the BYOD feature is not providing the correct answers to certain queries.

When I input the same question into the Azure Cognitive Search Explorer, it successfully returns the correct matches. However, when I pose the same question to the BYOD feature, it responds with "I don't have information."

I've double-checked my data and it seems to be correctly formatted and uploaded. I'm not sure why the BYOD feature isn't able to retrieve the correct information.

Has anyone else encountered this issue? Any suggestions on how to troubleshoot and fix this problem would be greatly appreciated.

Ziggy Zulueta 490 Reputation points MVP

2023-07-05T01:21:40.97+00:00

I get the same issue as well actually. I created my own employee handbook from scratch that was 30 pages long and uploaded it on the Cognitive Search index along with other documents. When I ask the question "what are the my employee benefits?" it will say "I don't have information." But when I ask the question "We have an employee handbook with the benefits stated there right?" and then it starts to churn out the answers.

I reckon that the service will not be perfect at the start and the system still needs to be trained over time to get the results we need.
romungi-MSFT 46,056 Reputation points Microsoft Employee

2023-07-06T14:04:06.4933333+00:00

@Kaja Sherief Referring similar question from SO for additional context.
Kaja Sherief 95 Reputation points

2023-07-07T10:11:59.98+00:00

@romungi-MSFT , Yes I have activated Semantic Search. However, when I pose the same query multiple times, I often receive the response, "I can't find any information." when I use the Azure Search Explorer with the same query, it provides the correct matching answer. Yet, when I use the same query with my own data, it fails to deliver a response.
YutongTie-MSFT 51,741 Reputation points

2023-07-10T17:18:52.0633333+00:00

@Kaja Sherief Thanks for your response, we are investigating this issue internally.
YutongTie-MSFT 51,741 Reputation points

2023-07-11T17:44:00.5733333+00:00

@kaja sherief Could you please provide a sample question to us so that we can bring this issue to the attention of product team?
SnehaAgrawal-MSFT 21,586 Reputation points

2023-07-12T05:47:00.4266667+00:00

@Kaja Sherief

After further discussion with PG on this it came out that the troubleshooting steps listed above, it’s not a fair 1:1 direct comparison as when you send a query to Cognitive Search directly (assuming with Semantic search turned on), it simply retrieves the most relevant documents using BM25 scoring algorithm and reranks the top 50 using the Semantic models from Bing (Semantic).

But when you type a query in BYOD feature in Azure OpenAI studio, what happens behind the scenes is that Cognitive Search retrieves the top “k” (where k is the top most relevant document count) documents and then passes in the content into the LLM prompt. The response you see from the LLM prompt (e.g GPT-4), is simply the output from your prompt and contextual data (via retrieved from Cognitive Search passed into the prompt).

This is why you may be seeing a difference in the results in the Azure Search Explorer vs BYOD.
Kaja Sherief 95 Reputation points

2023-07-12T06:39:44.6433333+00:00

@SnehaAgrawal-MSFT

Thank you for the detailed explanation on the difference between the workings of Cognitive Search and the BYOD feature in Azure OpenAI studio. I understand now why there might be a discrepancy in the results.

However, I'm still encountering an issue with the BYOD feature. I have uploaded HR documents into the blob, created the data source, skillset, indexer, and index in Azure Search, and then configured the BYOD to refer to the created index. When I ask questions like "explain internet policy" or "explain leave policy", the BYOD feature responds with "I don't have information."

Could you please guide me on how to troubleshoot and resolve this issue? Your assistance would be greatly appreciated.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Pinson, Noah 15 Reputation points

2023-07-12T17:22:44.9133333+00:00

@SnehaAgrawal-MSFT

I am using the same features within Azure and am having similar problems, even with a small, clean, test dataset. Is there a way that I can increase the top "k" documents when querying the bot?
SnehaAgrawal-MSFT 21,586 Reputation points

2023-07-13T04:32:31.7033333+00:00
@Kaja Sherief

Could you please reach out to us by sending an email with subject line “Attn:Sneha |thread title” to AzCommunity[at]microsoft[dot]com referencing this thread, for deeper investigation as below information is required.

Were the same questions on the same index working before? When did you start seeing the unexpected results?

Do you get "I don't have information." type responses for all questions now, or some?

Did you appropriately map the index data fields as prompted in the Studio?

Please share the index, that would greatly help to debug.
Balazs Nemeth 10 Reputation points

2023-08-22T15:23:16.2366667+00:00

@SnehaAgrawal-MSFT What API you suggest to call to avoid the cognitive search misbehave you explained above? The URL the OAI Studio provides does not return the proper answer, seemingly it does not considers your BYOD data at all.

I mean this url:
Balasubramanian Ramanathan 5 Reputation points

2023-09-11T13:00:49.5+00:00

I am also having the same issue for more than 1 month.
Ziggy Zulueta 490 Reputation points MVP

2023-09-12T16:23:38.4566667+00:00

Yes I have the same issue. Do we have any updates on this?
JessrylShingRabago-6578 0 Reputation points

2024-04-09T07:45:18.4666667+00:00

Same issue here. Any updates? I am assuming when using semantic ranker, if the top "k" is 5, should it be the top 5 out of top 50 results of semantic ranker being used for generating response? If yes, why still responded with no information even if the results from Cognitive Search is exactly among the top 5?

1 answer

Balazs Nemeth 10 Reputation points

2023-08-22T15:14:03.36+00:00
Have you guys resolved this issue? I'm facing the same.

In the OpenAI editor (https://oai.azure.com/ ) it gives the correct answer,

If you do an App Service deployment in the OAI editor (image below), it does provide the correct answers - for a few days...

If you call your BYOD deployment through Cognitive API it also works for a few days, but after it stops...

The dedicated endpoint the OAI editor offers (images below) does not considers the BYOD data at all...

I appreciate any ideas.

Creating your custom App Service deployment for testing:

Getting your custom endpoint:
Please sign in to rate this answer.
Pedram Meskoubhaghighi 5 Reputation points

2023-08-30T16:02:25.5+00:00

the solution to this issues is that the BYOD has another endpoint. You need to add extensions/chat... in ur url. Checkout this page, try out the curl in ur terminal for faster testing and validation. https://learn.microsoft.com/en-us/azure/ai-services/openai/use-your-data-quickstart?tabs=command-line&pivots=rest-api#example-curl-commands
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure OpenAI 'Bring Your Own Data' Feature Not Providing Correct Answers

1 answer

Your answer