Azure Cognitive Services Speech to Text get Confidence Scores during Recognizing Event

Technicator 5 Reputation points

I'm building a phone chatbot that's using Azure's Speech To Text service, and while it's good, one missing feature I would like is getting confidence scores for partial transcriptions during the Recognizing callback:

Right now, I can only can get confidence scores during the Recognized event, via e.result.json.NBest. This is after I set the speechConfig outputFormat to 1 and setting wordLevelConfidence to true for service property.

The reason I need the confidence scores during Recognizing event is that we want to detect ASAP when the user is speaking so we can interrupt the phone chatbot mid-sentence. Without confidence scores, it's difficult to know when the user is actually speaking or the speech to text service is just picking up background noise (e.g. radio or ambient conversation noises)

Azure AI Speech
Azure AI Speech
An Azure service that integrates speech processing into apps and services.
1,146 questions
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
1,878 questions
{count} vote

1 answer

Sort by: Most helpful
  1. dupammi 1,210 Reputation points Microsoft Vendor

    Hi @Technicator ,

    Thank you for reaching out to Microsoft Q&A with your question regarding confidence scores in Azure's Speech to Text service.

    You are on the right track. To obtain confidence scores for recognized events, you should configure the SpeechConfig with the appropriate output format. Specifically, set OutputFormat to OutputFormat.Detailed. Once configured, you can access the best possible recognitions, including the Confidence Score, using the Result.Best() method.

    please refer the following Speech recognition samples that can help you.

    Dealing with non-speech noise can be challenging. According to official documentation, it's recommended to ensure the user tries again or uses better recording conditions to avoid recognition of noise as speech. If this cannot be avoided, you can base it off the confidence score, but there is no guidance on the limit or cutoff. Depending on the quality of the speech, you could decide to ignore text below a certain threshold and can determine when to interrupt the chatbot.

    Here are some Speech recognition samples that can help you:

    Hope this helps.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.