How do adjust prosody in multilingual Text to speech?

Adrian Fiorito 20

I'm using Azure Text to speech to generate audio with SSML.
I want to be able to generate audio for sentences like:

<lang xml:lang="en-US">
Now, let's use "water" in a sentence. "I want some water" is: </lang>
<prosody rate="-20%">
  <lang xml:lang="ja-JP">水が欲しいです</lang>
</prosody>

Where the japanese word is read slower.

However, according to https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup-voice#adjust-speaking-languages,

"The <lang xml:lang> element is incompatible with the prosody and break elements. You can't adjust pause and prosody like pitch, contour, rate, or volume in this element."

Is there any work around or some other way to make the Japanese voice speak slower?

dupammi 7,745 Microsoft Vendor

Hi @Adrian Fiorito ,

Thanks for reaching out to us.

You can use the SSML to change the pause or add break -

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup-structure#add-a-break

The mstts:prosody element can be used inside the <voice> element to adjust the speaking rate of the Japanese text by "-20%"
https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup-voice#adjust-prosody

you can leverage SSML to change the style of the voice as well, as mentioned in the document here -

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup-voice#use-speaking-styles-and-roles

Please refer below code snippet, where the <voice> element is used to specify the Japanese voice with the name "Microsoft Server Speech Text to Speech Voice (ja-JP, Ayumi, Apollo)" and the English voice with the name "Microsoft Server Speech Text to Speech Voice (en-US, Jessa24kRUS)":

import os
import azure.cognitiveservices.speech as speechsdk


speech_key, service_region = "AZURE_SPEECH_KEY", "AZURE_SERVICE_REGION"

# Set the speech recognition language
speech_config.speech_recognition_language = "en-US"
#set the audio output config
audio_config = speechsdk.audio.AudioOutputConfig(use_default_speaker=True)

# Create a speech configuration with your subscription key and service region
speech_config = speechsdk.SpeechConfig(subscription=speech_key, region=service_region)
# Create a SpeechSynthesizer object
speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)

# Synthesize the SSML
ssml = """
<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xmlns:mstts="http://www.w3.org/2001/mstts" xml:lang="en-US">
  
  <voice name="Microsoft Server Speech Text to Speech Voice (en-US, Jessa24kRUS)">
    <lang xml:lang="en-US">
      Now, let's use "water" in a sentence. "I want some water" is:
    </lang>
  </voice>
  
  <voice name="Microsoft Server Speech Text to Speech Voice (ja-JP, Ayumi, Apollo)" style="cheerful">
    <mstts:prosody rate="-20%"><break time='200ms'/><!-- Custom pause duration --> 
      <lang xml:lang="ja-JP">水が欲しいです</lang>
    </mstts:prosody>
  </voice>
</speak>
"""
result = speech_synthesizer.speak_ssml_async(ssml).get()

Make sure to replace the placeholders - "AZURE_SPEECH_KEY", "AZURE_SERVICE_REGION" accordingly.

I hope this helps.

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics. Thank you!

Adrian Fiorito 20

Thanks so much for the help, that does indeed work as I was hoping.

Just noting down a strange issue I stumbled upon when doing this with the voice en-US-RyanMultilingualNeural:
The japanese part is said twice.

When the input is

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:mstts="http://www.w3.org/2001/mstts" xml:lang="en-US">
<voice name="en-US-RyanMultilingualNeural">
  <lang xml:lang="ja-JP">
    <mstts:prosody rate="-20%">
      あなたの夢は何ですか？
    </mstts:prosody>
   <break time='1s' />
  </lang>
</voice>
<voice name="en-US-RyanMultilingualNeural">
  <lang xml:lang="en-US">
   . Try saying it out loud: 
  </lang>
</voice>
</speak>

This does not occur if I remove the break or change to en-US-JennyMultilingualNeural. For me issue is resolved by moving the break into the english part

dupammi 7,745 Reputation points Microsoft Vendor

2023-10-16T12:36:59.88+00:00
Hi @Adrian Fiorito ,

The issue you encountered with the text being repeated only when using the "en-US-RyanMultilingualNeural" voice profile could be attributed to how the Text-to-Speech engine handles different voice profiles and their associated prosody and pause instructions.

Different voice profiles may have varying behaviors and interpretations of SSML elements. Some voice profiles may be more sensitive to the placement of elements like <break>, especially when used in conjunction with language switches or prosody changes.

It's possible that the "en-US-RyanMultilingualNeural" voice profile has a different SSML processing behavior compared to the "en-US-JennyMultilingualNeural" voice profile. This could explain why the issue occurred only with the former.

The placement of the <break> tag matters in SSML, and its behavior can be influenced by its context within the document. It affects the surrounding text segment based on its position in the SSML structure.

To resolve this issue, you've correctly removed the <break> tag. This confines the pause instruction doesn't affect the Japanese part.

Another solution that I reproduced here is, if you move the <break> tag inside the mstts:prosody tag, it becomes a part of the prosody instructions and doesn't act as a global pause.

i.e., It works for both voices, when the input is:

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xmlns:mstts="http://www.w3.org/2001/mstts" xml:lang="en-US"> <voice name="en-US-RyanMultilingualNeural"> <lang xml:lang="ja-JP"> <mstts:prosody rate="-20%"><break time='1s' /> あなたの夢は何ですか？ </mstts:prosody> </lang> </voice> <voice name="en-US-RyanMultilingualNeural"> <lang xml:lang="en-US"> . Try saying it out loud: </lang> </voice> </speak>

I hope this information helps. Thanks!
dupammi 7,745 Reputation points Microsoft Vendor

2023-10-17T11:38:25.11+00:00

Hi @Adrian Fiorito ,

Following up to see if the above answer was helpful. And, if you have any further query do let us know.
Adrian Fiorito 20 Reputation points

2023-10-17T11:50:36.54+00:00

Yes, thank you very much!

Accepted answer

dupammi 7,745 Reputation points Microsoft Vendor

2023-10-17T12:00:08.6666667+00:00

Hi @Adrian Fiorito ,

Thank you for reaching out to Microsoft Q&A.

Following up to see my above "comment" answer for the weird issue of "Japanese part being said twice", helps by checking the comments section of this thread. Do let us know if you have any queries.

To reiterate the resolution here, let me jot down the gist of my comment answer for your second question.

To resolve this issue of voice being told twice, you've correctly removed the <break> tag. This confines the pause instruction doesn't affect the Japanese part.

Another solution that I repro here is, if you move the <break> tag inside the [mstts:prosody] tag, it becomes a part of the prosody instructions and doesn't act as a global pause.

For a working sample python code and documentation, please refer comments section of this thread.

Please 'Accept as answer' and ‘Upvote’ if it helped, so that it can help others in the community looking for help on similar topics.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

How do adjust prosody in multilingual Text to speech?

0 additional answers