How to train Custom Speech-To-Text Model to recognize a word and capitalize the first letter of the phrase.

arvind prajapati 5

Hi, I have created a custom speech-to-text model that recognizes a phrase. It's recognizing it perfectly, but I want the resulting text in capitalized form. For E.g. the phrase is "Terminator" and the resulting text is "terminator" but I want the resulting text as "Terminator". The "T" of "terminator" should be a capital letter.
How should I train the model for it? I can do it manually by replacing the string, but I want to do it in a systematic way.

Thank you.

VasaviLankipalle-MSFT 14,181 Reputation points

2023-01-20T01:41:14.1233333+00:00
Hi @arvind prajapati , Thanks for using Microsoft Q&A Platform.

According to the Text normalization rules, the translation result appears to have:

Lowercase letters

Remove all punctuation within words except apostrophes.

Expand numbers, such as dollar amounts, into words or spoken form.

Will check back internally and provide you more information.

Regards,
Vasavi
arvind prajapati 5 Reputation points

2023-01-20T05:26:10.7266667+00:00

thanks for the response. Actually, I want the same result as when we speak "Microsoft" and "Google", the Speech-To-Text automatically capitalizes the first letter of both words. That's what I want to do whenever someone speaks the custom word I define in the model, the transcribed word's first letter should be in capital letters.
arvind prajapati 5 Reputation points

2023-01-24T05:49:44.5433333+00:00

@romungi-MSFT can you help me with this also?

romungi-MSFT 41,961 Microsoft Employee

Have you tried using the detailed output format. With the base model endpoints, the detailed output provides different formats like Lexical, ITN, MaskedITN and Display. I am not sure if the custom endpoints also provide one, but you can check by adding a simple setting with the speech config.

speech_config.output_format = speechsdk.OutputFormat.Detailed
speech_recognizer = speechsdk.SpeechRecognizer(
        speech_config=speech_config, language="en-US", audio_config=audio_config)
result = speech_recognizer.recognize_once()

From the result:

    if result.reason == speechsdk.ResultReason.RecognizedSpeech:
        print("Recognized: {}".format(result.text))

        # Time units are in hundreds of nanoseconds (HNS), where 10000 HNS equals 1 millisecond
        print("Offset: {}".format(result.offset))
        print("Duration: {}".format(result.duration))

        # Now get the detailed recognition results from the JSON
        json_result = json.loads(result.json)

        # The first cell in the NBest list corresponds to the recognition results
        # (NOT the cell with the highest confidence number!)
        print("Detailed results - Lexical: {}".format(json_result['NBest'][0]['Lexical']))
        # ITN stands for Inverse Text Normalization
        print("Detailed results - ITN: {}".format(json_result['NBest'][0]['ITN']))
        print("Detailed results - MaskedITN: {}".format(json_result['NBest'][0]['MaskedITN']))
        print("Detailed results - Display: {}".format(json_result['NBest'][0]['Display']))

VasaviLankipalle-MSFT 14,181 Reputation points

2023-01-24T22:25:00.0366667+00:00

Hi @arvind prajapati , as Rohit mentioned you can try the detailed output format.

If you are using Custom Speech model in Speech studio and looking for formatting output. We can still achieve that. As, I mentioned previously, for training the model it normalizes the text to these formatting rules. When the model is tested it comes with the formatted text. Please refer to the Custom speech model to know more about how to train and deploy the model.

I have used custom speech model and my deployed model shows the formatting output.

Please try these and let us know if you have any further questions.

I hope this helps.

Regards,
Vasavi
VasaviLankipalle-MSFT 14,181 Reputation points

2023-01-27T13:30:37.5733333+00:00

Hi @arvind prajapati , just checking in to see whether you require any additional assistance in this matter. Thanks!
VasaviLankipalle-MSFT 14,181 Reputation points

2023-02-03T01:53:05.48+00:00

@arvind prajapati , thanks for sharing this with us. I hope the issue is resolved.

1 answer

arvind prajapati 5 Reputation points

2023-02-04T23:06:13.8766667+00:00
@VasaviLankipalle-MSFT Sorry, for the late response. I was on leave these days. I went through @romungi-MSFT 's comment and found that the "NBest" property of Speech Recognition Result is not available in Golang SDK, so I can't access the "NBest" property. So I have to manipulate the data manually.

Update: I just found that how can I get the "NBest" property from the result. The detailed output doesn't comes directly in The Result struct in Golang. It comes when the output format is Detailed and RequestWordLevelTimestamps() is called from the speech config variable.

speechRecognizer.Recognized(func(event speech.SpeechRecognitionEventArgs) { detailedOutput := event.Result.Properties.GetProperty(common.SpeechServiceResponseJSONResult, "") })

Thanks for the help.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment