SSML: Using <lang xml:lang=""> within a multilingual voice sounds incorrect / unlike when used with the language-specific voice

mkb13 11

I am developing a TTS application that pronounces "nonsense words" with specific language pronunciations. For example, I am using Polish language voices to pronounce non-Polish words. If I use a Polish-specific language, I hear what I expect (the words read as a Polish speaker would read them). However, if I use an English multilingual voice and then add <lang xml:lang="pl-PL">...</lang> within the voice tag, it seems to be disregarding the lang tag and just guessing the language. Since the words are not actually Polish, the pronunciation is all over the place.

In short examples below, I use the word "mij" with Andrew and Ava and I specify that they should be pronounced as Polish (should be pronounced like English "me"). Instead, for Andrew it sounds like the system is guessing it is English, and pronounces it like English "Midge", and for Ava it seems to guess it is Spanish and pronounces the "j" like English "h".

Andrew:

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xmlns:mstts="https://www.w3.org/2001/mstts" xml:lang="en-US">

<voice name="en-US-AndrewMultilingualNeural">

<lang xml:lang="pl-PL">mij</lang>

<lang xml:lang="en-US"> hello</lang>

</voice>

</speak>

Ava:

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xmlns:mstts="https://www.w3.org/2001/mstts" xml:lang="en-US">

<voice name="en-US-AvaMultilingualNeural">

<lang xml:lang="pl-PL">mij</lang>

<lang xml:lang="en-US"> hello</lang>

</voice>

</speak>

I think I did the format correctly -- I don't see any difference between these and the example found in the "Lang examples" section at https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup-voice#multilingual-voices-with-the-lang-element. (I just included the English word "hello" to show how I am mixing the languages.)

How can I make the multilingual voice honor the language I select with the lang tag? Edit: I just realized this also happens with real words, including a case that used to work (valid Turkish embedded in an English multilingual voice). Am I doing something wrong with the syntax? Does the <lang> tag not do anything anymore?

dupammi 7,140 Reputation points Microsoft Vendor

2024-05-06T04:10:28.55+00:00
Hi @mkb13

Thank you for using the Microsoft Q&A forum.

It seems like the issue might be related to the usage of the <lang> tag within a multilingual voice block. You were trying to use a specific language pronunciation within a multilingual voice context, but it's not being honored as expected.

By referring this supported languages document, I tried with below ssml, by using the Polish voice "pl-PL-ZofiaNeural" in one voice tag, "en-US-AndrewMultilingualNeural" in another voice tag and specifying Polish text within both, which works correctly. I also experimented with <lang> tag by specifying en-US for word "hello". It seems to work correctly for me.

# Synthesize the SSML ssml = """ <speak xmlns="http://www.w3.org/2001/10/synthesis" xmlns:mstts="http://www.w3.org/2001/mstts" xmlns:emo="http://www.w3.org/2009/10/emotionml" version="1.0" xml:lang="en-US"> <voice name="en-US-AndrewMultilingualNeural"> <lang xml:lang="pl-PL">Jeśli prowadzimy nauczanie domowe, najlepiej jest dostosować się do tego, co każdy dzień przynosi, i starać się dobrze się bawić po drodze. Dobrym początkiem jest wypróbowanie wielu aplikacji edukacyjnych, które pomagają dzieciom być szczęśliwymi i doskonale radzić sobie ze szkolnictwem jednocześnie.</lang> <lang xml:lang="en-US"> hello</lang> </voice> <voice name="pl-PL-ZofiaNeural"> <lang xml:lang="pl-PL">Jeśli prowadzimy nauczanie domowe, najlepiej jest dostosować się do tego, co każdy dzień przynosi, i starać się dobrze się bawić po drodze. Dobrym początkiem jest wypróbowanie wielu aplikacji edukacyjnych, które pomagają dzieciom być szczęśliwymi i doskonale radzić sobie ze szkolnictwem jednocześnie.</lang> <lang xml:lang="en-US"> hello</lang> </voice> <voice name="en-US-AvaMultilingualNeural"> <mstts:audioduration value="15s"/> hello </voice> </speak> """

I request you to experiment with different voices and adjust the text accordingly to achieve the desired pronunciation. That said, spelling individual words, without using them in a sentence will produce some discrepancies in language, speech models. Please experiment with sentences to see if it works. Due to limitation of files that can be attached on this thread, I am unable to upload the audio files for the above inputs. However, it is working fine for me.

I hope this helps. Thank you.
mkb13 11 Reputation points

2024-05-06T06:49:11.66+00:00

Thanks, but did you try my examples? "mij" should sound like "me", not "midge" or "mih" (and I would expect it to sound the same for both male and female voices). I am using Polish to pronounce a language that is not supported by Azure TTS. I want it to pronounce the words as a Polish person would if they saw them. Maybe it needs to be a non "neural" voice - are any of those still available? For the Turkish case, I had Ava say "Hoş geldiniz?" in Turkish surrounded by English, and the "o" was pronounced wrong (like English instead of Turkish).
dupammi 7,140 Reputation points Microsoft Vendor

2024-05-06T07:24:07.7366667+00:00

Hi @mkb13 ,

I suggest you try with sentences rather than individual words. Trying with individual words, alone, without using them in a sentence will produce some discrepancies in the speech models. Please experiment with sentences to see if it works.

while using multilingual voices, the system attempts to recognize and pronounce multiple languages accurately, but it might not always fully honor specific language tags, especially for pronunciation nuances. As a workaround, you might need to use a Polish-specific voice for better pronunciation accuracy or try using the Phonetic alphabets which are used with the Speech Synthesis Markup Language (SSML) to improve the pronunciation of text to speech voices. Speech service supports the International Phonetic Alphabet (IPA)

For more info: SSML phonetic alphabets & Structured text phonetic pronunciation data

Hope this helps! Thank you.
mkb13 11 Reputation points

2024-05-08T04:20:22.4166667+00:00

The same thing happens with multiple words - I included only one word to show the minimal repro case. I am trying to get the voice to pronounce words as if they were being read by a Polish person. Can you please tell me how to make this work? Are there any other voices available (old style, maybe?) that can do this?
dupammi 7,140 Reputation points Microsoft Vendor

2024-05-08T07:02:55.93+00:00

Hi @mkb13

Sorry for the trouble you are facing. I request you to raise a support case through Azure portal.This will allow you to get assistance from Azure support in resolving the issue you're facing. Please provide a detailed description of the issue, including any error messages or screenshots, when raising the support case.

Share via

SSML: Using <lang xml:lang=""> within a multilingual voice sounds incorrect / unlike when used with the language-specific voice