Text to Speech failed to output some format with background audio

Leo Malfray 65

I'm using a Speech Service to do Text to Speech (called from a javascript azure function).

I use SSML with the mstts:backgroundaudio tag where the source (src) is an mp3 audio file.

When I output as mp3 (Audio16Khz32KBitRateMonoMp3) or Pcm wav (Riff16Khz16BitMonoPcm) using SpeechSynthesisOutputFormat no problem, it works as expected.

However when I choose a MULAW or ALAW output format (Raw8Khz8BitMonoALaw, Raw8Khz8BitMonoMULaw, Riff8Khz8BitMonoALaw, Riff8Khz8BitMonoMULaw) I get an internal server error on your side.

Works fine without the background audio tag.

Was able to work around by generating in PCM and converting to MULAW with an external library but that add an extra step that should not be necessary if it was working as expected on your side.

Not sure if there is another place for issue reporting ?

navba-MSFT 27,465 Reputation points Microsoft Employee

2024-01-10T06:30:33.4+00:00
@Leo Malfray Welcome to Microsoft Q&A Forum, Thank you for posting your query here!
.
Could you please check and test from the Speech Studio if you are facing the same issue ?
.
Are you using the most recent version of Microsoft Cognitive Services Speech SDK for JavaScript in your application ?

.
Could you also enable the JS SDK logging from your application and check if there are any errors / exceptions being logged:

sdk.Diagnostics.SetLoggingLevel(sdk.LogLevel.Debug); sdk.Diagnostics.SetLogOutputPath("LogfilePathAndName");

. If none of the above plan helps, Please share your sample code along with the text input you had used, so that I can test this at my end. . Awaiting your reply.
Leo Malfray 65 Reputation points

2024-01-10T07:24:44.42+00:00
In the Speech studio you can't precisely choose the export audio format like when using code but yet I can reproduce the error:

<speak version="1.0" xml:lang="en-US" xmlns:mstts="http://www.w3.org/2001/mstts"> <mstts:backgroundaudio src="https://file-examples.com/storage/fe5f588707659e1c5986414/2017/11/file_example_MP3_700KB.mp3" volume="0.7" fadein="3000" fadeout="4000"/> <voice name="en-US-JennyNeural">The text provided in this document will be spoken over the background audio.</voice> </speak>

just choose 8kHz for the sample rate and you'll get a 500 Internal server error

(that ssml code is from your documentation, just changed the backgroundaudio src with a valid random mp3 example found online)

If you want to test with code I am pretty sure you already have a test environment, whatever the language is, just use this previous SSML code and try to change the "SpeechSynthesisOutputFormat" https://learn.microsoft.com/en-us/dotnet/api/microsoft.cognitiveservices.speech.speechsynthesisoutputformat?view=azure-dotnet to anything like MULAW or ALAW (for example "Riff8Khz8BitMonoMULaw") and you will get an Internal server error, pretty easy to reproduce.
navba-MSFT 27,465 Reputation points Microsoft Employee

2024-01-10T12:19:31.81+00:00

@Leo Malfray I am able to reproduce this issue at my end using the 8khz sample rate from speech studio. I am checking this further. I will get back to you once I have an update on this.
navba-MSFT 27,465 Reputation points Microsoft Employee

2024-01-12T08:31:34.0666667+00:00

@Leo Malfray I looked at the backend logs and also involved the Product Group team to look into this. Seems like the 8khz rate is unsupported. So, Until the pipelines are updated to use the new elements, the only data you should be sending is signed 16-bit little endian PCM with the sample rate set to either 16 KHz or 24 KHz. Any other data will result in a pipeline parse error (InternalServerError). They are checking if this can be documented as unsupported until then. Hope this clarifies. Please let me know if you have any follow-up questions.
Leo Malfray 65 Reputation points

2024-01-12T09:45:00.11+00:00

Your explanation is not clear, you're talking about me sending something?

I think what you want to say is that at the moment outputting in 8kHz while using and audio background tag is not working. Or is there scenarii where it could works ? with specific source as background audio ?

Even if I found a work around outputting in 16kHz and then converting in 8kHz using another library this is quite unsatisfying.

Your answer is quite similar to the one google was giving when their text to speech API was not working as documented: "ho yeah it does not work, deal with it" instead of trying to fix your mistakes, and that's exactly why I was considering moving to your services instead (and now I'm no longer sure about that).

What is, and yes my question is very serious here, what is your recruitment process to work on the team in charge with that product ? Do you use external contractors ?
I'm a software engineer with experience as senior lead developer and would be very interested in fixing that specific issue, at a discount price ;)
navba-MSFT 27,465 Reputation points Microsoft Employee

2024-01-15T04:22:30.62+00:00

@Leo Malfray Thanks for getting back. Please accept our sincere apologies for the inconvenience caused. We have escalated this issue to the Product Group team. I will update this thread once I hear from them.
navba-MSFT 27,465 Reputation points Microsoft Employee

2024-01-22T06:00:49.28+00:00

@Leo Malfray We are still looking into this issue. We will keep you posted about the progress. We appreciate your patience on this.

Share via

Text to Speech failed to output some format with background audio

Your answer