How can I successfully stream Azure TTS audio from my server to the client using Fetch API and PassThrough push stream?

RL_Anon 5 Reputation points
2024-01-06T06:32:57.38+00:00

I am attempting to stream Azure TTS from my server to the client using the Fetch API and a PassThrough push stream. The expected outcome is to receive the stream in chunks. The actual output is a response object with no information. I have tried creating a ReadableStream using Fetch, but when I try to log the response, I get an error message that my response object is size 0. I have also tried to see if anything is getting sent through in chunks, but everything is size 0. I have tried to debug my backend but from everything I can tell, it is working properly. If anyone has solved this issue or has demo code for streaming TTS in JavaScript, please let me know. This is my actual function code. I believe it works.:

const generateSpeechFromText = async (text) => {
  const speechConfig = sdk.SpeechConfig.fromSubscription(
    process.env.SPEECH_KEY,
    process.env.SPEECH_REGION
  );
  speechConfig.speechSynthesisVoiceName = "en-US-JennyNeural";
  speechConfig.speechSynthesisOutputFormat =
    sdk.SpeechSynthesisOutputFormat.Audio16Khz32KBitRateMonoMp3;

  const synthesizer = new sdk.SpeechSynthesizer(speechConfig);

  return new Promise((resolve, reject) => {
    synthesizer.speakTextAsync(
      text,
      (result) => {
        if (result.reason === sdk.ResultReason.SynthesizingAudioCompleted) {
          const bufferStream = new PassThrough();
          bufferStream.end(Buffer.from(result.audioData));
          resolve(bufferStream);
        } else {
          console.error("Speech synthesis canceled: " + result.errorDetails);
          reject(new Error("Speech synthesis failed"));
        }
        synthesizer.close();
      },
      (error) => {
        console.error("Error in speech synthesis: " + error);
        synthesizer.close();
        reject(error);
      }
    );
  });

This is my index.js route code to send to the frontend. I believe it works but there could be an error.:

app.get("/textToSpeech", async (request, reply) => {
  if (textWorks) {
    try {
      const stream = await generateSpeechFromText(
        textWorks
      
      );
      console.log("Stream created, sending to client: ", stream);
      reply.type("audio/mpeg").send(stream);
    } catch (err) {
      console.error(err);
      reply.status(500).send("Error in text-to-speech synthesis");
    }
  } else {
    reply.status(404).send("OpenAI response not found");
  }
});

This is my frontend client code. I think the error has to do with the response object, but I am not sure.:

// Fetch TTS from Backend
export const fetchTTS = async (): Promise
Azure AI Speech
Azure AI Speech
An Azure service that integrates speech processing into apps and services.
2,069 questions
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
3,621 questions
{count} vote

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.