[Bug] Dispatch eval command gives wrong scores in the Summary.html

Question

When testing a DIspatch-App, we would like to use the command "dispatch eval" to use the provided statistics and information sheets. Unfortunately, especially the sheets "misclassified" and "Low Confidence" don't give correct information when testing the utterances directly in the LUIS Portal. Example: utterance "abc" is labeled as misclassified, but actually works perfectly fine Is this bug known, because we would really like to use the statistics but unfortunately they seem to be rather useless ☹

Answer 1

Hi YutongTie,

thanks for your quick reply.

Unfortunately I'm not a developer/software engineer and only work with LUIS within the LUIS portal and Dispatch CLI, but I'll try to explain our problem with some screenshots.

1. LUIS Portal
Let's say I'll use utterance "ich habe eine kontaktanfrage" (our LUIS model is in German) which is one of our example utterances in our app SmalltalkSkill, which works perfectly fine:

2. Summary.html
When evaluating the Dispatch app via "dispatch eval" in the Dispatch CLI, I get the following results in the sheet "Misclassified" (from the Summary.html)

The SmalltalkSkill app is correctly labeled as "Predicted Intent" in the LUIS Portal, but in the Summary.html we see the AdvancePaymentSkill as predicted intent/app - as we have seen before, the utterance isn't actually misclassified, the Summary.html seems to be wrong.

3. Comparision between scores in the LUIS Portal and the scores Sheet "Low Confidence"

Here is also a big difference between the scores shown in the portal and the scores shown in the Summary.html

I hope you can understand my problem now, basically I would like to use the "dispatch eval" commands, but the results shown in the Summary.html seem to be wrong.

Thank you for your answer in advance!

Answer 2

I'm still waiting for an answer ☹