Speech Recognition not producing good results

Question

Hi everyone,

I have been trying to setup an API using Python to make real-time speech recognition.

Context

I am using azure.cognitiveservices.speech library. I followed the tutorial here, using continuous recognition.

Basically, my API receives an audio stream with WebSocket protocol. The received buffer is pushed into a stream (azure.cognitiveservices.speech.audio.PushAudioInputStream). This stream is used in the AudioConfig.

Problem

The code seems to be working. However, the obtained transcript is not accurate and the result is pretty bad. I don't know why. The received stream is in PCM format, 32 bits, 1 channel, 48 Hz. I checked the integrity of the data by saving an audio file: nothing to report, the audio is perfectly audible and of good quality.

Code

This is the main code that I am using. The Transcription class is instanciated when the WebSocket connection opens, then the audio data is pushed thanks to the push_data() method, in a loop.

import azure.cognitiveservices.speech as speechsdk

speech_key = os.environ.get('SPEECH_KEY')
speech_region = os.environ.get('SPEECH_REGION')
language = "fr-FR"


class Transcription:
    def __init__(self):
        speech_config = speechsdk.SpeechConfig(subscription=speech_key, region=speech_region)
        speech_config.speech_recognition_language=language

        # setup the audio stream
        audio_format = speechsdk.audio.AudioStreamFormat(samples_per_second=48000, bits_per_sample=32, channels=1)
        self.stream = speechsdk.audio.PushAudioInputStream(audio_format)
        audio_config = speechsdk.audio.AudioConfig(stream=self.stream)

        # instantiate the speech recognizer with push stream input
        self.speech_recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config)

        # Connect callbacks to the events fired by the speech recognizer
        self.speech_recognizer.recognizing.connect(lambda evt: print('RECOGNIZING...: {}'.format(evt)))
        self.speech_recognizer.recognized.connect(lambda evt: print('RECOGNIZED: {}'.format(evt)))
        self.speech_recognizer.session_started.connect(lambda evt: print('SESSION STARTED: {}'.format(evt)))
        self.speech_recognizer.session_stopped.connect(self.stop)
        self.speech_recognizer.canceled.connect(self.stop)

    def start(self):
        self.speech_recognizer.start_continuous_recognition()

    def stop(self, evt="forced stop"):
        print('CLOSING on {}'.format(evt))
        self.stream.close()
        self.speech_recognizer.stop_continuous_recognition()

    def push_data(self, frames):
        self.stream.write(frames)

Call-to-help

Do you have any idea of where this problem comes from? I would gladly take any suggestion to help me through this issue.

Thank you in advance.

Answer 1

Solution proposée et validée par @Emmanuella Michel

Après avoir manipulé les données audio pour changer le format, j'ai trouvé la solution.

L'audio que j'utilise est au format suivant : non-interleaved IEEE754 32-bit linear PCM with a nominal range between -1 and +1, that is, a 32-bit floating point buffer, with each sample between -1.0 and 1.0. Il s'agit en fait du flux contenu dans l'objet AudioBuffer de l'API Web Audio. Comme vous pouvez le constater, j'ai bien spécifié ce format dans AudioStreamFormat. Cependant, un détail que la documentation d'Azure ne souligne pas (je n'ai pas trouvé de mention de cela), c'est qu'Azure Speech SDK attende un encodage en entiers. Or, les données audio dont je dispose sont encodés en flottants (Float32Array en Javascript). Il faut donc convertir les données en utilisant un encodage en entiers. La signature sur 32 bits et la fréquence d'échantillonnage de 48 kHz peuvent être conservés, tant que cela est bien spécifié dans AudioStreamFormat. La conversion en entiers a résolu le problème et la transcription fonctionne très bien désormais. Je trouve que la documentation d'Azure mérite d'être plus claire sur les formats audio compatibles. En espérant que cela puisse en aider d'autres !

Answer 2

I have a thread here solving the issue, if you wanna take a look, feel free to comment the post in StackOverflow: https://stackoverflow.com/a/78743136/26354907