PVA Chabot with Azure Language Studio

Edirin Imiruaye 5

I built a chatbot with PVA and connected it to azure language studio custom question and answering to manage the knowledge base - a simple excel document with question and answer columns and about 150 rows. I've a few had cases of it not being able to return accurate answers and rather that failing, ends up providing a totally wrong answer.

I've trained and re-trained the model as best i can with alternate questions but it seems like it's a bit to intelligent for its own good and tries too hard to get things right and gives a wrong answer in the end.

does anyone have any tips that might help?

romungi-MSFT 42,206 Reputation points Microsoft Employee

2023-10-27T06:14:56.37+00:00
@Edirin Imiruaye If you are seeing a wrong answer along with a low confidence score you should consider adding a confidence threshold value in your request so that the response provides a valid answer if the value is above the input threshold. This should just be a parameter in the body of the request. For ex: Below request uses a threshold limit of 0.95 to eliminate any incorrect responses. This also means that the default answer of the project might be the response for most of the questions if accuracy is the main criteria.

curl -X POST -H "Ocp-Apim-Subscription-Key: $LANGUAGE_KEY" -H "Content-Type: application/json" -d '{ "question": "How much battery life do I have left?", "confidenceScoreThreshold": "0.95", }' '$LANGUAGE_ENDPOINT.api.cognitive.microsoft.com//language/:query-knowledgebases?projectName=Sample-project&api-version=2021-10-01&deploymentName={DEPLOYMENT_NAME}'

Since you have mentioned that the project is retrained, did you deploy the project again as an endpoint and then try to use with your bot? Thanks!!
Edirin Imiruaye 5 Reputation points

2023-10-31T13:17:43.5+00:00

@romungi-MSFT please could you provide some additional details. Cant seem to find where to enter this code.

Allow me provide a bit more details on my bot and hopefully you can guide me. Bot is built in PVA and has a few topics in it. I've edited the fallback topic so that it includes a power automate flow that connects to azure custom question and answering where the knowledge base is.

The only place i can find anything related to a confidence score is withing azure when i'm testing my bot.
romungi-MSFT 42,206 Reputation points Microsoft Employee

2023-11-01T13:21:04.19+00:00

@Edirin Imiruaye I am not familiar with power automate flows, The above suggestion will work only if you are able to add the additional parameter in the body of the request. If the flow connection does not allow to add this then it might only be using the question in the request.
Amée Björkdahl, Johannes 0 Reputation points

2024-04-19T11:36:05.6866667+00:00

@romungi-MSFT
When trying to import the Conversational Language Project to Copilot Studio it seems like the settings object in the JSON is totally ignored which leads to the confidenceThreshold not being imported.

I cannot find any settings within the Analyze Text topic either that references any confidence or threshold which makes me think that this is not supported in the Copilot Studio.

Can you or someone confirm if this is supported or not? I've been on a wild goose chase trying to somehow handle low confidence answers without adding too much to the language model itself (e.g. adding random phrases to the "None" intent so it's caught there instead of predicting intents with very low confidence scores).