Which Azure Speech SDK Feature to Use for Real-Time Meeting Transcription with Speaker Diarization?

Hi Su Myat Hlaing,

Did you get any chance to check the above response. Thank you!

Hi @Amira Bedhiafi , @Anonymous ,

Thanks for your response. I have already implemented the recommended method (audio preprocessing), but I am still experiencing significant issues with real-time transcription:

Issues Faced:

Speaker Diarization is Inaccurate

• There are 5 speakers, but the SDK only recognizes 2 and incorrectly groups multiple speakers under the same ID.
  • Different people are being assigned the same speaker ID, making it impossible to differentiate who is speaking.
  Audio Preprocessing Doesn’t Seem to Work
  • I have applied gain control, noise suppression, and bandpass filtering using Web Audio API before passing the audio to Azure Speech SDK.
  • However, recognition accuracy has not improved—speech is still missed or misrecognized.
  • Could there be an issue with my implementation? Is there another way to improve audio quality before transcription?
  Recognition Accuracy Varies Across Devices
  • On some PCs, recognition is significantly worse, even when using the same setup.
  • Are there hardware dependencies (e.g., microphone quality, CPU performance) affecting accuracy?
  Code Implementation:

speechConfig.setProperty(SpeechSDK.PropertyId.SpeechServiceConnection_SpeakerIdMode, "True");

I also apply audio preprocessing before feeding the microphone input into the Speech SDK:

const audioContext = new AudioContext();
const mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = audioContext.createMediaStreamSource(mediaStream);

const gainNode = audioContext.createGain();
gainNode.gain.value = 5; // Amplify microphone input

const biquadFilter = audioContext.createBiquadFilter();
biquadFilter.type = "bandpass"; 
biquadFilter.frequency.setValueAtTime(2500, audioContext.currentTime);

const destination = audioContext.createMediaStreamDestination();
source.connect(gainNode).connect(biquadFilter).connect(destination);

const audioConfig = SpeechSDK.AudioConfig.fromStreamInput(destination.stream);

Even with these enhancements, the system still fails to recognize speech accurately.

Questions:

1. How can I ensure the SDK assigns unique speaker IDs to all speakers?
2. Why is my audio preprocessing not working? Is my code incorrect? Is there a better way?
3. Why does recognition accuracy vary across different devices? Are there specific hardware requirements?
4. Is using a custom model my last option? Would training a custom model significantly improve diarization and recognition?

I would appreciate any guidance on how to resolve these issues.

Thanks!

Hi Su Myat Hlaing,

To ensure that the Azure Speech SDK assigns unique speaker IDs to all speakers, you should ensure that each participant has a voice signature created in advance. This is crucial for accurate speaker identification. If some speakers are not recognized, it may be due to the absence of their voice samples in the system. You can collect audio recordings from each user and send them to the signature generation service to create user profiles for speaker identification.

Regarding your audio preprocessing, while applying gain control, noise suppression, and bandpass filtering can help improve audio quality, the effectiveness can vary based on the implementation and the specific characteristics of the audio input. It's important to ensure that the preprocessing is correctly applied and that the microphone setup meets the specifications required for optimal performance.

As for the variation in recognition accuracy across different devices, hardware dependencies such as microphone quality and CPU performance can significantly impact the transcription accuracy. Using a high-quality microphone and ensuring that the device meets the recommended specifications can help mitigate these issues.

If you find that the built-in capabilities are still not meeting your needs, exploring the option of using a custom model could be beneficial. Training a custom model may improve both diarization and recognition, especially if your use case involves specific vocabulary or speaker characteristics that are not well captured by the default model.

please refer this Real-time conversation transcription multichannel diarization (preview).

Thank you!

Hi @SriLakshmi C ,
Custom models can only be trained with mono-channel audio, as noted in the documentation. If I train the Azure Speech model with a single user's voice (mono audio), will the model perform better for that user specifically? And if so, for different users, would they need to undergo training beforehand to achieve optimal recognition? Thanks in advance for your insights!

Hi Su Myat Hlaing,

Training the Azure Speech model with a single user's voice using mono-channel audio can indeed improve recognition accuracy for that specific user. This is because custom models can be tailored to recognize the unique characteristics of an individual's speech patterns and pronunciation.

For different users, they would also need to undergo training with their respective voice samples to achieve optimal recognition. Each user's voice has distinct features, and training the model with their audio data would enhance its ability to accurately transcribe their speech.
Thank you!