question

PeterTan-4296 avatar image
0 Votes"
PeterTan-4296 asked PeterTan-4296 answered

QnA Maker confidence score changes even with the same question - change due to context?

Hi.

In our chatbot, we experience the answer's confidence score of the same question changes - and it seems to depend on the timing it was asked.

If it is asked first time from our home menu, the score is consistent.

If the same question is asked after a certain reply, the score we get is lower.
- the observation I got is that, this happens if the reply of the previous question has related questions

Can I ask if this is the expected behavior - theoretically I think it is, but I hope it can be confirmed as it helps on our design of the threshold score and multi-turn conversations.

May I ask if there are other factors that can affect the score too?

Thank you.





azure-qna-maker
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello,

Could you please share the testing data you are using for this? This will help us to test and investigate. Thanks a lot.


Regards,
Yutong

0 Votes 0 ·

1 Answer

PeterTan-4296 avatar image
0 Votes"
PeterTan-4296 answered

Hi Yutong,

Thanks for the response. I confirmed the behavior in the following test using the Test function in QnA Maker - it is easy to replicate (not just the Generate Answer API with the answers value).

I quickly created in QnA Maker with following Q-A Pairs
- a question with single turn answer
- a question with an answer with related questions


You can see the image attached the score went down after asking the same question right after a reply with related questions. Is there a way to keep the score through some parameters in API call or configuration?

84753-screen-shot-2021-04-06-at-151205.png



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.