Implementation of realtime STT using external stream.

Question

Implementation of realtime STT using external stream.

ywmo 41

Hello, I am trying to implement spring server that uses STOMP protocol
recieving stream of data , and sends it to azure speech to get STT result.
Currently I am stuck in sending the audio stream to azure. According to the log of my subscription, the recieved audio file is empty. can you give me some advice for it? Below is the code implementation.

(FYI : external stream is from web audio API, and has followed the audio stream condtion suggested below: https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-use-audio-input-streams )

In spring server the process goes like this:

create SpeechConfig from my subscription
create pushStream which is instance of PushAudioInputStream, and create audioConfig using pushStream

   PushAudioInputStream pushStream  = AudioInputStream.createPushStream();
   AudioConfig audioConfig = AudioConfig.fromStreamInput(pushStream);

start speech recognition

   recognizer.startContinuousRecognitionAsync().get();
   ...

create buffer and read the audioInputstream then write it on the pushStream (audioInputStream is a ByteArrayInputStream)

   while (true){   
   if (!((bytesRead = audioInputStream.read(readBuffer)) != -1)) break;    
   if (bytesRead == readBuffer.length){        
           pushStream.write(readBuffer);    
}   
   else    {   pushStream.write(Arrays.copyOfRange(readBuffer, 0, bytesRead));    }
}

stop speech recognition and close the resources.

romungi-MSFT 48,771 Reputation points Microsoft Employee

2023-02-24T11:07:44.9433333+00:00
ywmo There is a good working example of using the PushAudioStream for speech recognition on this page.

The only difference in your code compared to the sample is using audioInputStream() instead of FileInputStream()

If you are seeing no data or input, then I would add additional logs to check if any bytes are read. Also, try to use an InputStream with a wav file as in this sample to confirm if the stream data is actually written.

InputStream inputStream = new FileInputStream("YourAudioFile.wav"); . . . // Push audio read from the file into the PushStream. while ((bytesRead = inputStream.read(readBuffer)) != -1) { if (bytesRead == readBuffer.length) { pushStream.write(readBuffer); } else { // Last buffer read from the WAV file is likely to have less bytes pushStream.write(Arrays.copyOfRange(readBuffer, 0, bytesRead)); } }

I hope this helps to debug the issue with your audioConfig. Thanks!!
romungi-MSFT 48,771 Reputation points Microsoft Employee

2023-02-28T07:12:52.4033333+00:00

ywmo Did you get a chance to check if above suggestion helped?
ywmo 41 Reputation points

2023-03-08T06:34:50.7933333+00:00

Thanks for your fast reply( I was away from this problem for week, so I couldn't apply your advice)

I have managed to send stream from web API to Azure and get audio file in the log but all of them sounded wierdly compare to the original voice. (The stream from normal file worked well)

I believe there is some error in formatting before sending the data from web audio API, because web Audio API gives float32 Array so data preprocessing to Int16 Array is needed. Do you have any idea for formatting in web audio API? , any possible mistakes?

and Also would STOMP protocol be appropriate for implementing Real Time STT?
Thanks in advance.
romungi-MSFT 48,771 Reputation points Microsoft Employee

2023-03-08T07:34:15.8866667+00:00

Thanks @ywmo for confirming that audio file stream worked well. I am not quite sure about the web audio format supported with the speech SDK. As you mentioned it might need some processing, but I am not aware of how this can be done. Posting an issue on the speech SDK repo will help confirm if this is supported.

Your answer

romungi-MSFT 48,771 Reputation points Microsoft Employee

2023-02-24T11:07:44.9433333+00:00

ywmo There is a good working example of using the PushAudioStream for speech recognition on this page.

The only difference in your code compared to the sample is using audioInputStream() instead of FileInputStream()

If you are seeing no data or input, then I would add additional logs to check if any bytes are read. Also, try to use an InputStream with a wav file as in this sample to confirm if the stream data is actually written.

InputStream inputStream = new FileInputStream("YourAudioFile.wav"); . . . // Push audio read from the file into the PushStream. while ((bytesRead = inputStream.read(readBuffer)) != -1) { if (bytesRead == readBuffer.length) { pushStream.write(readBuffer); } else { // Last buffer read from the WAV file is likely to have less bytes pushStream.write(Arrays.copyOfRange(readBuffer, 0, bytesRead)); } }

I hope this helps to debug the issue with your audioConfig. Thanks!!
romungi-MSFT 48,771 Reputation points Microsoft Employee

2023-02-28T07:12:52.4033333+00:00

ywmo Did you get a chance to check if above suggestion helped?
ywmo 41 Reputation points

2023-03-08T06:34:50.7933333+00:00

Thanks for your fast reply( I was away from this problem for week, so I couldn't apply your advice)

I have managed to send stream from web API to Azure and get audio file in the log but all of them sounded wierdly compare to the original voice. (The stream from normal file worked well)

I believe there is some error in formatting before sending the data from web audio API, because web Audio API gives float32 Array so data preprocessing to Int16 Array is needed. Do you have any idea for formatting in web audio API? , any possible mistakes?

and Also would STOMP protocol be appropriate for implementing Real Time STT?
Thanks in advance.
romungi-MSFT 48,771 Reputation points Microsoft Employee

2023-03-08T07:34:15.8866667+00:00

Thanks @ywmo for confirming that audio file stream worked well. I am not quite sure about the web audio format supported with the speech SDK. As you mentioned it might need some processing, but I am not aware of how this can be done. Posting an issue on the speech SDK repo will help confirm if this is supported.

Share via

Implementation of realtime STT using external stream.

Your answer