QnA Maker confidence score changes even with the same question - change due to context?

asked 2021-03-31T03:27:27.033+00:00
Peter Tan 96 Reputation points

Hi.

In our chatbot, we experience the answer's confidence score of the same question changes - and it seems to depend on the timing it was asked.

If it is asked first time from our home menu, the score is consistent.

If the same question is asked after a certain reply, the score we get is lower.

  • the observation I got is that, this happens if the reply of the previous question has related questions

Can I ask if this is the expected behavior - theoretically I think it is, but I hope it can be confirmed as it helps on our design of the threshold score and multi-turn conversations.

May I ask if there are other factors that can affect the score too?

Thank you.

Azure QnA Maker
Azure QnA Maker
An Azure Cognitive Service for Language feature that distills information into conversational answers.
138 questions
{count} votes

Accepted answer
  1. answered 2021-04-06T06:16:38.717+00:00
    Peter Tan 96 Reputation points

    Hi Yutong,

    Thanks for the response. I confirmed the behavior in the following test using the Test function in QnA Maker - it is easy to replicate (not just the Generate Answer API with the answers value).

    I quickly created in QnA Maker with following Q-A Pairs

    • a question with single turn answer
    • a question with an answer with related questions

    You can see the image attached the score went down after asking the same question right after a reply with related questions. Is there a way to keep the score through some parameters in API call or configuration?

    84753-screen-shot-2021-04-06-at-151205.png

    No comments

0 additional answers

Sort by: Most helpful