Azure Speech SDK - Listen to multiple (agent and customer) audio streams

Arun Srinivasan 70

We are using Azure Speech SDK for speech recognition (microsoft-cognitiveservices-speech-sdk: 1.35.0). We are able to set up speechConfig, audioConfig and are able to recognize the text both in live and batch mode. During the live mode, the Audio config is set to default microphone input. It is able to seek the microphone input well using the below command. speechsdk.AudioConfig.fromDefaultMicrophoneInput())

async sttFromMic() {
      const tokenObj = await getTokenOrRefresh();      
      const speechConfig = speechsdk.SpeechConfig.fromAuthorizationToken(tokenObj.authToken, tokenObj.region);
      speechConfig.speechRecognitionLanguage = 'en-US';         
      const audioConfig = speechsdk.AudioConfig.fromDefaultMicrophoneInput();
      recognizer = new speechsdk.SpeechRecognizer(speechConfig, audioConfig);
....

Assume it is a conversation between Agent and Customer. The agent microphone input stream is being listened and recognized using the above code. But how do we listen to the customer audio stream? Kindly advise.

navba-MSFT 20,810 Reputation points Microsoft Employee

2024-02-12T03:35:02.76+00:00

@Arun Srinivasan Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

Real-time diarization enables conversations to be transcribed in real-time while simultaneously identifying speakers. Diarization refers to the ability to tell who spoke and when. It differentiates speakers in mono channel audio input based on their voice characteristics. This allows for the identification of speakers during conversations and can be useful in a variety of scenarios such as doctor-patient conversations, agent-customer interactions, and court proceedings.

Please note that Real-time diarization is currently in public preview.

The java script sample code for Create real-time diarization is available here:

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-stt-diarization?tabs=windows&pivots=programming-language-javascript

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
navba-MSFT 20,810 Reputation points Microsoft Employee

2024-02-16T05:55:42.7633333+00:00

@Arun Srinivasan Just following up to check if my suggestion helped. Please let me know if you have any further queries. I would be happy to help.
Arun Srinivasan 70 Reputation points

2024-02-18T08:43:16.6666667+00:00
Thanks for your response. This is not about identifying the speaker. We have hosted a web app which identifies the speech using Azure speech SDK. Assume the below scenario.

Opened the web app on windows chrome /edge browser.

Click on start button which listens to default microphone using Azure speech SDK. const audioConfig = speechsdk.AudioConfig.fromDefaultMicrophoneInput();

And assume I'm in teams call with another person, it is able to recognize only my voice and not the other person. My voice is recognized correctly into text but when others speak it is not able recognize their voices (Referring to speech text from other person)

But strangely the behavior is different on a Mac. When the webapp is opened on Safari / Chrome browser on Mac, It recognizes the speech of both myself and the other person in teams call.

Not sure why is this happening? My question is, the speech to text happens with the primary persons microphone and not the other person in the call on Windows OS.

Dont know if this is got to do with OS based configuration or something else?
navba-MSFT 20,810 Reputation points Microsoft Employee

2024-02-21T10:59:14.9266667+00:00

@Arun Srinivasan Thanks for your reply.

Please note that the Real-time diarization identifies the speaker and also the text that they spoke (Speech to text). The java script sample code for Create real-time diarization is available here: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-stt-diarization?tabs=windows&pivots=programming-language-javascript

Regarding your question about OS dependencies, The Speech SDK supports the following languages and platforms:

More info **here. **Hope this helps.
Arun Srinivasan 70 Reputation points

2024-02-28T02:57:32.34+00:00

This is not about diarization (identification of voices). I'm trying to run realtime speech to text transcription on two streams (one through mic, one through speaker). Using Azure audioconfig audioConfig = speechsdk.AudioConfig.fromDefaultMicrophoneInput(); I'm able to listen to microphone and able to transcribe the speech in real time. But I'm not able to transcribe the speaker voice. Can you help with this? For reference there was another thread in 2021 which is dated now. Hoping there is a resolution for this now in 2024. Thanks in advance for your help. https://learn.microsoft.com/en-us/answers/questions/445472/speech-to-text-on-two-streams.
navba-MSFT 20,810 Reputation points Microsoft Employee

2024-02-28T04:45:27.03+00:00

@Arun Srinivasan Thanks for clarifying.

To transcribe the speaker’s voice, you would need to capture the speaker’s audio output as an input stream to the Azure Speech Service. This can be a bit tricky because it involves routing the speaker’s audio output back into an audio input stream.

Here’s a general approach to solve this:
Arun Srinivasan 70 Reputation points

2024-02-28T05:33:52.22+00:00

Can you detail out the approach please?
navba-MSFT 20,810 Reputation points Microsoft Employee

2024-02-28T07:20:41.26+00:00
@Arun Srinivasan Apologies, somehow my complete message didn't go through in my previous comment.

Adding on to my above comment. here is the detailed approach:

Capture the Speaker’s Audio Output: You would need to use a “virtual audio cable” or similar software that can create a “virtual microphone” that listens to your speaker output. There are several third-party software options available for this, such as VB-Cable: VB-Audio Virtual Apps (vb-audio.com). After that, select the microphone linked with the speaker as the default microphone. And then, you can run a SpeechRecognizer with input from a microphone, following the function RecognitionWithLanguageAndDetailedOutputAsync in cognitive-services-speech-sdk/speech_recognition_samples.cs at master · Azure-Samples/cognitive-services-speech-sdk (github.com)

Transcribe the Captured Audio: Once you have the speaker’s audio output being captured by the virtual microphone, you can then use the Azure Speech SDK to transcribe this audio just like you are doing with the actual microphone input.

Share via

Azure Speech SDK - Listen to multiple (agent and customer) audio streams