Azure OpenAI Whisper hallucinates source audio language

Question

Azure OpenAI Whisper hallucinates source audio language

Shamus Sim 20

Hi,

I have been experimenting with the Whisper service via Azure OpenAI services.

There are alot of cases where the audio language send to the API is (clealry) English but it comes out as a Malay(Bahasa Malaysia) text, despite clearly using the transcription API and not the translation API.

Sample prompt and API call

            `transcription = client_tts.audio.transcriptions.create(`
````                    model="whisper",`

`                    file=audio_file,`

`                    prompt="""Transcribe the following audio file, it is taken in medical clinic setting, `

`                    return the original language and the transcription. Only return English, Bahasa Malaysia, and Mandarin language"""`

`                )`

SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-02-27T12:04:28.5533333+00:00

Hello Shamus Sim,

Greetings and Welcome to Microsoft Q&A!

I understand that you're encountering an issue with Whisper's language detection incorrectly identifying the source language. Please try these steps:

Try explicitly specifying the language in your API request by setting the "language" parameter to "en" for English.

Ensure that the audio quality is clear, as background noise or accents can sometimes affect detection accuracy. If the problem persists, consider testing with different audio samples and checking if the issue is consistent across multiple requests.

Try segmenting the audio into smaller parts before sending it to the API. This can help reduce errors caused by long or complex audio inputs.

Experiment with different Whisper models, as some versions may handle language detection and transcription more accurately than others. Testing various approaches can help identify the best configuration for your specific use case.

I hope you understand. And, if you have any further query do let us know.

Thank you!
SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-03-03T09:08:52.1133333+00:00

Hi Shamus Sim,

Did you get any chance to check the above response. Thank you!
SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-03-04T14:47:40.4133333+00:00

Hi Shamus Sim,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thank you!

Your answer

SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-02-27T12:04:28.5533333+00:00

Hello Shamus Sim,

Greetings and Welcome to Microsoft Q&A!

I understand that you're encountering an issue with Whisper's language detection incorrectly identifying the source language. Please try these steps:

Try explicitly specifying the language in your API request by setting the "language" parameter to "en" for English.

Ensure that the audio quality is clear, as background noise or accents can sometimes affect detection accuracy. If the problem persists, consider testing with different audio samples and checking if the issue is consistent across multiple requests.

Try segmenting the audio into smaller parts before sending it to the API. This can help reduce errors caused by long or complex audio inputs.

Experiment with different Whisper models, as some versions may handle language detection and transcription more accurately than others. Testing various approaches can help identify the best configuration for your specific use case.

I hope you understand. And, if you have any further query do let us know.

Thank you!
SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-03-03T09:08:52.1133333+00:00

Hi Shamus Sim,

Did you get any chance to check the above response. Thank you!
SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-03-04T14:47:40.4133333+00:00

Hi Shamus Sim,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thank you!

Share via

Azure OpenAI Whisper hallucinates source audio language

Your answer