How to get the JSON output that is shown in Speed Studio

Question

How to get the JSON output that is shown in Speed Studio

Ritvij Sharma 60

User's image

I want this JSON object in my react project when I use the Azure Speech SDK. How can I get this?

The issue I have with the JS SDK is that the word level timestamps are not accurate, some offset-duration combos has 2 words. I need exact word by word durations at least. And I only see that here in this JSON object.

What is the right solution to get this word by word durations?

navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-23T07:23:12.8533333+00:00

@Ritvij Sharma Welcome to Microsoft Q&A Forum, Thank you for posting your query here!
I understand that you are looking for a solution to get word-by-word durations using the Azure Speech SDK. You have mentioned that the word level timestamps are not accurate and you need exact word-by-word durations.

Please note that the similar issue was reported in the past. The fix has been added for this issue. See this comment: https://github.com/microsoft/cognitive-services-speech-sdk-js/issues/394#issuecomment-874973957

Another GitHub thread for the similar issue is here.

The most recent version of the SDK should have the fix for this. Could you please ensure that you have updated your SDK to the most recent version and then test it again ?

Hope this helps.
Ritvij Sharma 60 Reputation points

2023-11-23T08:05:01.71+00:00
@navba-MSFT Hello. I checked and yes I have the latest version of the package.

I would also like to mention that in the recognizing event, this is the JSON output I get through console.log:

RECOGNIZING: Text= { "Id":"42d4437a42d440b384a8c97704320999", "Text":"content is king", "Offset":700000, "Duration":10400000, "PrimaryLanguage": { "Language":"en-US" }, "Channel":0 }

As you can see, the JSON gives text which has 3 words, not word-by-word output.

Please tell me how to proceed from this point, and how I can get exact word-by-word output.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-23T08:27:12.6533333+00:00

@Ritvij Sharma Thanks for confirming.

To request the offset and duration per word, first you must set the corresponding SpeechConfig property as shown here:

speechConfig.requestWordLevelTimestamps();

So, could you please call the requestWordLevelTimestamps() method ? This includes word-level timestamps. This was added in version 1.7.0.

More info here and here.

Ritvij Sharma 60

I have added that here is the code for initializing the speech recognizer:

import {
  AudioConfig,
  CancellationDetails,
  CancellationReason,
  ResultReason,
  SpeechConfig,
  SpeechRecognizer,
} from "microsoft-cognitiveservices-speech-sdk";

const speechConfig = SpeechConfig.fromSubscription(
  "key",
  "region"
);
speechConfig.speechRecognitionLanguage = "en-US";

export async function fromFile() {
  console.log("fetching file blob");
  let response = await fetch("/sample-mp3.wav");
  let data = await response.blob();
  let metadata = {
    type: "audio/wave",
  };

  const file = new File([data], "sample-mp3.wav", metadata);
  console.log("done");
  let audioConfig = AudioConfig.fromWavFileInput(file);
  try {
    console.log("requesting word level timestamps");
    speechConfig.requestWordLevelTimestamps();
  } catch (error) {
    console.log(`error: ${error}`);
  }

  let speechRecognizer = new SpeechRecognizer(speechConfig, audioConfig);
  return speechRecognizer;
}

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Ritvij Sharma 60 Reputation points

2023-11-23T09:22:29.1733333+00:00

@navba-MSFT my last comment above has the correct code. Please check that.

navba-MSFT 27,540 Microsoft Employee Moderator

@Ritvij Sharma Thanks for the details. From you code, could you please invoke this REST API for Speech to test and check. You should get the response in below format for every word with its offset and duration:

{
  "RecognitionStatus": "Success",
  "Offset": "400000",
  "Duration": "11000000",
  "NBest": [
      {
        "Confidence" : "0.87",
        "Lexical" : "good morning",
        "ITN" : "good morning",
        "MaskedITN" : "good morning",
        "Display" : "Good morning.",
        "PronScore" : 84.4,
        "AccuracyScore" : 100.0,
        "FluencyScore" : 74.0,
        "CompletenessScore" : 100.0,
        "Words": [
            {
              "Word" : "Good",
              "AccuracyScore" : 100.0,
              "ErrorType" : "None",
              "Offset" : 500000,
              "Duration" : 2700000
            },
            {
              "Word" : "morning",
              "AccuracyScore" : 100.0,
              "ErrorType" : "None",
              "Offset" : 5300000,
              "Duration" : 900000
            }
        ]
      }
  ]
}

Ritvij Sharma 60 Reputation points

2023-11-23T09:54:51.8866667+00:00

@navba-MSFT The API you have shared is from the "Speech to text REST API for short audio" document, which says in the beginning that it can take only 60 seconds of audio. In my project, I am expecting bigger files, too, like up to 15-20 minutes.

Any workaround for that?
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-24T10:19:53.9066667+00:00

@Ritvij Sharma Thanks for getting back. I am sharing 2 more action plans for bigger files.

Plan 1:
Please test using the sample code shared here by ensuring that "wordLevelTimestampsEnabled": true:
https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/batch/js/node/main.js

Plan 2:
You can test from Speech Studio using Batch speech to text as shown below:

You can also get the sample curl REST API command:

Please try to invoke this rest api from the code.

I am also sharing the REST API link here if that helps.

https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-2-preview1/operations/Transcriptions_Create

Hope this helps.
Ritvij Sharma 60 Reputation points

2023-11-24T13:05:50.12+00:00

Thank you for your help @navba-MSFT !
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-25T05:29:40.0833333+00:00

@Ritvij Sharma Thanks for your reply. Please do not forget to "Accept the answer” and “up-vote” the below answer, so that this can be beneficial to other community members.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-25T05:30:03.9766667+00:00

@Ritvij Sharma Thanks for your reply. Please do not forget to "Accept the answer” and “up-vote” the above answer, so that this can be beneficial to other community members.
Ritvij Sharma 60 Reputation points

2023-12-01T04:36:28.9233333+00:00

The correct answer to this, without having to do batch transcriptions and using the JS SDK itself, is to use pronunciation assessment. On activating pronunciation assessment, the results give the offset and duration for each and every word in the speech.

Related documentation to activate pronunciation assessment is here: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-pronunciation-assessment?pivots=programming-language-javascript

Accepted answer

0 additional answers

Your answer

navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-23T07:23:12.8533333+00:00

@Ritvij Sharma Welcome to Microsoft Q&A Forum, Thank you for posting your query here!
I understand that you are looking for a solution to get word-by-word durations using the Azure Speech SDK. You have mentioned that the word level timestamps are not accurate and you need exact word-by-word durations.

Please note that the similar issue was reported in the past. The fix has been added for this issue. See this comment: https://github.com/microsoft/cognitive-services-speech-sdk-js/issues/394#issuecomment-874973957

Another GitHub thread for the similar issue is here.

The most recent version of the SDK should have the fix for this. Could you please ensure that you have updated your SDK to the most recent version and then test it again ?

Hope this helps.
Ritvij Sharma 60 Reputation points

2023-11-23T08:05:01.71+00:00

@navba-MSFT Hello. I checked and yes I have the latest version of the package.

I would also like to mention that in the recognizing event, this is the JSON output I get through console.log:

RECOGNIZING: Text= { "Id":"42d4437a42d440b384a8c97704320999", "Text":"content is king", "Offset":700000, "Duration":10400000, "PrimaryLanguage": { "Language":"en-US" }, "Channel":0 }

As you can see, the JSON gives text which has 3 words, not word-by-word output.

Please tell me how to proceed from this point, and how I can get exact word-by-word output.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-23T08:27:12.6533333+00:00

@Ritvij Sharma Thanks for confirming.

To request the offset and duration per word, first you must set the corresponding SpeechConfig property as shown here:

speechConfig.requestWordLevelTimestamps();

So, could you please call the requestWordLevelTimestamps() method ? This includes word-level timestamps. This was added in version 1.7.0.

More info here and here.
Ritvij Sharma 60 Reputation points

2023-11-23T09:15:23.3733333+00:00

I have added that here is the code for initializing the speech recognizer:

import { AudioConfig, CancellationDetails, CancellationReason, ResultReason, SpeechConfig, SpeechRecognizer, } from "microsoft-cognitiveservices-speech-sdk"; const speechConfig = SpeechConfig.fromSubscription( "key", "region" ); speechConfig.speechRecognitionLanguage = "en-US"; export async function fromFile() { console.log("fetching file blob"); let response = await fetch("/sample-mp3.wav"); let data = await response.blob(); let metadata = { type: "audio/wave", }; const file = new File([data], "sample-mp3.wav", metadata); console.log("done"); let audioConfig = AudioConfig.fromWavFileInput(file); try { console.log("requesting word level timestamps"); speechConfig.requestWordLevelTimestamps(); } catch (error) { console.log(`error: ${error}`); } let speechRecognizer = new SpeechRecognizer(speechConfig, audioConfig); return speechRecognizer; }
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Ritvij Sharma 60 Reputation points

2023-11-23T09:22:29.1733333+00:00

@navba-MSFT my last comment above has the correct code. Please check that.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-23T09:49:07.1133333+00:00

@Ritvij Sharma Thanks for the details. From you code, could you please invoke this REST API for Speech to test and check. You should get the response in below format for every word with its offset and duration:

{ "RecognitionStatus": "Success", "Offset": "400000", "Duration": "11000000", "NBest": [ { "Confidence" : "0.87", "Lexical" : "good morning", "ITN" : "good morning", "MaskedITN" : "good morning", "Display" : "Good morning.", "PronScore" : 84.4, "AccuracyScore" : 100.0, "FluencyScore" : 74.0, "CompletenessScore" : 100.0, "Words": [ { "Word" : "Good", "AccuracyScore" : 100.0, "ErrorType" : "None", "Offset" : 500000, "Duration" : 2700000 }, { "Word" : "morning", "AccuracyScore" : 100.0, "ErrorType" : "None", "Offset" : 5300000, "Duration" : 900000 } ] } ] }
Ritvij Sharma 60 Reputation points

2023-11-23T09:54:51.8866667+00:00

@navba-MSFT The API you have shared is from the "Speech to text REST API for short audio" document, which says in the beginning that it can take only 60 seconds of audio. In my project, I am expecting bigger files, too, like up to 15-20 minutes.

Any workaround for that?
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-24T10:19:53.9066667+00:00

@Ritvij Sharma Thanks for getting back. I am sharing 2 more action plans for bigger files.

Plan 1:
Please test using the sample code shared here by ensuring that "wordLevelTimestampsEnabled": true:
https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/batch/js/node/main.js

Plan 2:
You can test from Speech Studio using Batch speech to text as shown below:

You can also get the sample curl REST API command:

Please try to invoke this rest api from the code.

I am also sharing the REST API link here if that helps.

https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-2-preview1/operations/Transcriptions_Create

Hope this helps.
Ritvij Sharma 60 Reputation points

2023-11-24T13:05:50.12+00:00

Thank you for your help @navba-MSFT !
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-25T05:29:40.0833333+00:00

@Ritvij Sharma Thanks for your reply. Please do not forget to "Accept the answer” and “up-vote” the below answer, so that this can be beneficial to other community members.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2023-11-25T05:30:03.9766667+00:00

@Ritvij Sharma Thanks for your reply. Please do not forget to "Accept the answer” and “up-vote” the above answer, so that this can be beneficial to other community members.
Ritvij Sharma 60 Reputation points

2023-12-01T04:36:28.9233333+00:00

The correct answer to this, without having to do batch transcriptions and using the JS SDK itself, is to use pronunciation assessment. On activating pronunciation assessment, the results give the offset and duration for each and every word in the speech.

Related documentation to activate pronunciation assessment is here: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-pronunciation-assessment?pivots=programming-language-javascript

Answer 1

@Ritvij Sharma Thanks for your your reply.
.
I'm glad to see you were able to resolve your issue. Thanks for posting your solution so that others experiencing the same thing can easily reference this. Since the Microsoft Q&A community has a policy that the question author cannot accept their own answer, they can only accept answers by others, I'll repost your solution in case you'd like to Accept the answer.
.
Issue:
You are looking for a solution to get word-by-word durations using the Azure Speech SDK. You have mentioned that the word level timestamps are not accurate and you need exact word-by-word durations.
.
Resolution:
The correct answer to this, without having to do batch transcriptions and using the JS SDK itself, is to use pronunciation assessment. On activating pronunciation assessment, the results give the offset and duration for each and every word in the speech.

Related documentation to activate pronunciation assessment is here: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-pronunciation-assessment?pivots=programming-language-javascript

Share via

How to get the JSON output that is shown in Speed Studio

0 additional answers

Your answer