Is it possible to stream Groq LLM responses as and when I get it into Azure TTS?

arunnair 0

Hi! I'm trying to build a real time LLM conversation bot, and need it to be as low latency as possible. I have successfully set up TTS audio output streaming (https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-lower-speech-synthesis-latency?pivots=programming-language-python#streaming) using Azure's Python speech SDK, but would like to know if it's also possible to stream the input text to TTS, piece by piece, as and when I get it? So that I don't have to wait for entire text to be generated before generating the TTS audio.

I tried searching for it but didn't come across any relevant information.

dupammi 7,575 Reputation points Microsoft Vendor

2024-05-03T15:15:58+00:00
Hi @arunnair

Thank you for using the Microsoft Q&A platform.

By seeing your description, I understand that you are planning to Stream the Groq LLM responses to Azure Text-to-Speech (TTS) to reduce latency in your real-time conversation bot.

Refer a similar thread here. There was a sample shown in it, for the output streaming scenario. However, you can adapt it for input text streaming or as needed by your use case implementation.

Here's a basic outline of how this can be done:

Streaming: Utilize streaming to lower latency. Start playback when the first audio chunk is received. In a service scenario, forward audio chunks immediately to clients instead of waiting for the whole audio.

AudioDataStream: Use AudioDataStream from the Speech SDK to enable streaming.

Pre-connect and Reuse SpeechSynthesizer: Pre-connect to the Speech service when you anticipate needing the connection soon. Reuse SpeechSynthesizer instances to avoid connection latency.

Transmit Compressed Audio: Compressed audio formats help reduce latency, especially in unstable or low-bandwidth networks.

Here are general best practices, as per the official docs.

general-best-practices-to-mitigate-throttling-during-autoscaling

example-of-a-workload-pattern-best-practice

I hope the information detailed helps. Thank you.
dupammi 7,575 Reputation points Microsoft Vendor

2024-05-03T20:46:44.5533333+00:00

Hi @arunnair

We haven’t heard from you on the last response and was just checking back to see if you got a chance to see my above suggestions. Thank you.
arunnair 0 Reputation points

2024-05-04T10:19:40.67+00:00

@dupammi Yes I did go through that post before, but I didn't find the part where they are sending LLM response chunks into Azure. How do I do that?
dupammi 7,575 Reputation points Microsoft Vendor

2024-05-04T15:58:01.0066667+00:00
Hi @arunnair

Here are the high-level steps that you can implement in your application:

Set up the Azure Speech Service credentials and the Text-to-Speech client.

Define a custom Python function that establishes a WebSocket connection with Azure Speech Service, initializes a SpeechSynthesizer object with the WebSocket connection, and then sends LLM response chunks to Azure and synthesizes each chunk to speech using the synthesize_audio() method of the SpeechSynthesizer.

Please see this section on usage of wss private endpoint with custom domain name with the SDK. Also refer to my last response documentation for best practices.I hope you understand. Thank you.

Share via

Is it possible to stream Groq LLM responses as and when I get it into Azure TTS?