Issue with Pronunciation Assessment in Speech Recognition API Always Returning PronScore 100

Question

Issue with Pronunciation Assessment in Speech Recognition API Always Returning PronScore 100

Heba Ghazaly 5

I am using the https://eastus.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1 API with the POST method for speech-to-text conversion. Here are the details of my implementation:

Programming Language: JavaScript
Parameters:
- language
- format

Headers:

Pronunciation-Assessment: constructed as follows:


    const pronAssessmentParamsJson = `{"ReferenceText":"${currentWordTxt}","GradingSystem":"HundredMark","Dimension":"Comprehensive","Format":"Detailed"}`;

    const pronAssessmentParams = Buffer.from(pronAssessmentParamsJson, 'utf-8').toString('base64');

Content Type:
- 'Content-Type': 'audio/wav'
Body:
- Includes the audio blob.

Problem:

No matter the audio content, the API always returns PronScore: 100.0. For instance, if the ReferenceText is "citizen", the response always gives a PronScore of 100.0, regardless of whether the word "citizen" is pronounced correctly, incorrectly, or not at all.

Example:

ReferenceText: "citizen"
Audio Blob: A short recording of my pronunciation (or silence)
Response: A JSON object where PronScore is 100.0

Troubleshooting Efforts:

Tried the API on Postman separately from my JavaScript code, and I still get the same response.
No error codes were returned from the API.

Goal:

I aim to check if the audio matches the text for a student activity. If it doesn't match, I use the score from the API to add to the student's score progress.

Could anyone assist in identifying why the PronScore is always 100 and how to resolve this issue?

Thank you.

santoshkc 15,355 Reputation points Microsoft External Staff Moderator

2024-08-08T13:31:46.0566667+00:00
Hi @Heba Ghazaly,

Thank you for reaching out to Microsoft Q&A forum!

I tried to repro your issue using Postman using https://eastus.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US&format=detailed, but able to get the below response:

{ "RecognitionStatus": "Success", "Offset": 1100000, "Duration": 9200000, "NBest": [ { "Confidence": 0.8336352, "Lexical": "first floor", "ITN": "first floor", "MaskedITN": "first floor", "Display": "First floor." }

See: Speech to text Rest API.I recommend you to raise a support request through the Azure portal. They will be able to investigate the issue further and provide a more targeted solution. Thank you.
santoshkc 15,355 Reputation points Microsoft External Staff Moderator

2024-08-09T14:56:14.2666667+00:00

Hi @Heba Ghazaly,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Thank you.
Heba Ghazaly 5 Reputation points

2024-08-12T08:05:41.15+00:00

Hello @santoshkc ,

Thank you for your feedback.

To add some more context to my situation. The issue arises when I add a ReferenceText parameter to the Pronunciation-Assessment. The reference text is intentional because I want the API to compare it against the audio sent in the body to generate a PronScore. However, the problem is that I always receive a score of 100, regardless of the audio's accuracy.

Here's an example of a request. I attached an audio with .wav extension, with a ReferenceText provided in the Pronunciation-Assessment header.
The pronunciation assessment contains the reference text for a word other than "site," yet the API returned "site" as the recognized word with a PronScore of 100.
santoshkc 15,355 Reputation points Microsoft External Staff Moderator

2024-08-12T16:52:05.7033333+00:00
Hi @Heba Ghazaly,

Sorry for the trouble you are facing. I recommend reporting this issue to the Azure Speech Services support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Speech Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure Speech Services support team will review your request and provide assistance as soon as possible.
Mohammed Natour 0 Reputation points

2024-10-17T09:05:07.96+00:00

Heba Ghazaly was the issue resolved? I am facing a similar problem
Heba Ghazaly 5 Reputation points

2024-10-17T09:22:35.1766667+00:00

Mohammed Natour Unfortunately, I couldn't resolve the issue directly. However, I implemented a workaround by removing the ReferenceText parameter and avoiding the model comparison. Instead, now I utilize the model solely as a speech recognizer to capture the words. Then I perform the comparison manually with the reference text using the Levenshtein distance algorithm to calculate the Levenshtein ratio.

2 answers

Your answer

santoshkc 15,355 Reputation points Microsoft External Staff Moderator

2024-08-08T13:31:46.0566667+00:00

Hi @Heba Ghazaly,

Thank you for reaching out to Microsoft Q&A forum!

I tried to repro your issue using Postman using https://eastus.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US&format=detailed, but able to get the below response:

{ "RecognitionStatus": "Success", "Offset": 1100000, "Duration": 9200000, "NBest": [ { "Confidence": 0.8336352, "Lexical": "first floor", "ITN": "first floor", "MaskedITN": "first floor", "Display": "First floor." }

See: Speech to text Rest API.I recommend you to raise a support request through the Azure portal. They will be able to investigate the issue further and provide a more targeted solution. Thank you.
santoshkc 15,355 Reputation points Microsoft External Staff Moderator

2024-08-09T14:56:14.2666667+00:00

Hi @Heba Ghazaly,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Thank you.
Heba Ghazaly 5 Reputation points

2024-08-12T08:05:41.15+00:00

Hello @santoshkc ,

Thank you for your feedback.

To add some more context to my situation. The issue arises when I add a ReferenceText parameter to the Pronunciation-Assessment. The reference text is intentional because I want the API to compare it against the audio sent in the body to generate a PronScore. However, the problem is that I always receive a score of 100, regardless of the audio's accuracy.

Here's an example of a request. I attached an audio with .wav extension, with a ReferenceText provided in the Pronunciation-Assessment header.
The pronunciation assessment contains the reference text for a word other than "site," yet the API returned "site" as the recognized word with a PronScore of 100.
santoshkc 15,355 Reputation points Microsoft External Staff Moderator

2024-08-12T16:52:05.7033333+00:00

Hi @Heba Ghazaly,

Sorry for the trouble you are facing. I recommend reporting this issue to the Azure Speech Services support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Speech Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure Speech Services support team will review your request and provide assistance as soon as possible.
Mohammed Natour 0 Reputation points

2024-10-17T09:05:07.96+00:00

Heba Ghazaly was the issue resolved? I am facing a similar problem
Heba Ghazaly 5 Reputation points

2024-10-17T09:22:35.1766667+00:00

Mohammed Natour Unfortunately, I couldn't resolve the issue directly. However, I implemented a workaround by removing the ReferenceText parameter and avoiding the model comparison. Instead, now I utilize the model solely as a speech recognizer to capture the words. Then I perform the comparison manually with the reference text using the Levenshtein distance algorithm to calculate the Levenshtein ratio.

Answer 1

Hello Mohammed Natour and Heba Ghazaly,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you both are having similar issues where Pronunciation Assessment returning a score of 100.0 in all cases.

Before you contact support, based on the guidelines and best practices to resolve this issue, try the following check lists:

Incorrect audio formats can lead to unexpected results, make sure the audio blob sent in the request matches the required format specifications for the API and it's (must be) in WAV format (.wav) with proper encoding (e.g., PCM, 16-bit, 16kHz, mono).

Yes, you're using Base64 encoding for the Pronunciation Assessment parameters though, PronAssessmentParams required structured and encoded string in correctly formed and passed in the headers, here is an example:

      const pronAssessmentParamsJson = `{
          "ReferenceText": "${currentWordTxt}",
          "GradingSystem": "HundredMark",
          "Dimension": "Comprehensive",
          "Format": "Detailed"
      }`;
      const pronAssessmentParams = Buffer.from(pronAssessmentParamsJson).toString('base64');

The header should look like this snippet too:

   {
          'Pronunciation-Assessment': pronAssessmentParams,
          'Content-Type': 'audio/wav'
   }

Low-quality or silent audio can cause the API to default to high confidence or perfect scores, even when it shouldn't. Make sure it's high-quality audio you're using to ensure that the API can clearly detect speech.
The general speech-to-text endpoint may not provide proper pronunciation scoring, so you have to use the dedicated pronunciation assessment endpoint in Azure or include the correct settings in the headers. Check this link for limitations and characteristics: https://learn.microsoft.com/en-us/legal/cognitive-services/speech-service/pronunciation-assessment/characteristics-and-limitations-pronunciation-assessment

Therefore, if the API still returns inaccurate scores after verifying the above, consider opening a support request via the Azure portal for more investigation.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Mohammed Natour 0 Reputation points

2024-10-17T19:04:01.0733333+00:00

Hi Sina Salam, my problem is slightly different, the issue I'm facing happens when I try to make consecutive requests to the STT API, the first request works perfectly, but in the following requests, I receive the ReferenceText which I sent as the DisplayText. It is as if the result of previous requests is being cached by the API and relayed back to me, even though each request is sent with a different audio file.

P.S: The headers and audio format of my request are correct. As I said, the first API request works fine, the problem is with the following requests.

Thank you.
Sina Salam 22,031 Reputation points Volunteer Moderator

2024-10-17T19:26:10.2733333+00:00

Hello

Thank you for your feedback.

Regarding your new update: Check out the list of how you can get rid of the reoccurrences.

Disable any client-side caching: Ensure that your HTTP client isn’t caching responses. Use headers like Cache-Control: no-cache, no-store, must-revalidate or Pragma: no-cache.

Check the API settings: Some APIs might have caching mechanisms by default. Verify if there is an option to disable caching for requests or responses.

Ensure that each API request is unique. You can:

Add a unique identifier (e.g., GUID, timestamp) to each request or in the metadata section to force the API to treat it as a fresh request.

Check if the API allows specifying an ID per request to distinguish between them.

If you are maintaining a session or using any stateful connection between requests, ensure that the state is reset after each request. You could:

Reinitialize the SpeechConfig and PronunciationAssessmentConfig objects between consecutive requests.

Close and reopen the connection after each request to prevent reusing the same state.

If the issue persists, enable detailed logging on the API server side (if possible) to trace the exact request/response behavior and if you're on paid subscription, open a support request via the Azure portal.
Mohammed Natour 0 Reputation points

2024-10-17T19:46:49.3733333+00:00

Hello again, the issue was related to my network class. in my implementation, I would create a singleton instance that would be used throughout the application. The issue is resolved when I create a new network class instance for each request. I hope this helps someone <3

Answer 2

Hello Mohammed Natour,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer. Accepted answers show up at the top, resulting in improved discoverability for others.

Issue: Customer Pronunciation Assessment in Speech Recognition API Always Returning PronScore 100

Error Message: None.

Solution: Customer created a new network class instance for each request.

Share via

Issue with Pronunciation Assessment in Speech Recognition API Always Returning PronScore 100

2 answers

Your answer