QnA Maker confidence score changes even with the same question - change due to context?

Question

Hi.

In our chatbot, we experience the answer's confidence score of the same question changes - and it seems to depend on the timing it was asked.

If it is asked first time from our home menu, the score is consistent.

If the same question is asked after a certain reply, the score we get is lower.

• the observation I got is that, this happens if the reply of the previous question has related questions

Can I ask if this is the expected behavior - theoretically I think it is, but I hope it can be confirmed as it helps on our design of the threshold score and multi-turn conversations.

May I ask if there are other factors that can affect the score too?

Thank you.

Answer 1

Hi Yutong,

Thanks for the response. I confirmed the behavior in the following test using the Test function in QnA Maker - it is easy to replicate (not just the Generate Answer API with the answers value).

I quickly created in QnA Maker with following Q-A Pairs

• a question with single turn answer
• a question with an answer with related questions

You can see the image attached the score went down after asking the same question right after a reply with related questions. Is there a way to keep the score through some parameters in API call or configuration?