How to integrate Avaya Aura Experience Portal (AAEP) and UniMRCP server running with Microsoft Azure Speech Recognition and Speech Synthesis

Question

How to integrate Avaya Aura Experience Portal (AAEP) and UniMRCP server running with Microsoft Azure Speech Recognition and Speech Synthesis

Shahnawaz 41

Hello,
I am seeking advice on integration of Whisper Model into Avaya via UniMRCP Azure plugin for SPEECH to TEXT. May I know how can we make sure we are using the whisper model for ASR. Below is the documentation on integration but I am unable to understand how to achieve this. https://docs.unispeech.io/en/ums/avaya/azsr-azss

Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2024-01-11T11:30:34.0766667+00:00

@Shahnawaz I think in this case the installation and steps are related to 3rd party usage of Azure speech service, the documentation link above does not specify how this can be configured based on what I have read. This document though says that Azure speech can be configured for following endpoints. https://docs.unispeech.io/en/ums/ms-azure/speech-recog/install-rpm

The above does not specify if batch endpoint is supported, since whisper models are supported through batch transcription API, more details about Whisper model with Azure speech is documented here. I would recommend to also contact support of unispeech to confirm the same. Thanks!!
Shahnawaz 41 Reputation points

2024-01-12T00:58:04.6466667+00:00
Hello @Rohit Mungi , Thanks for the reply. While going through the document I found that in the configuration setting we can provide the endpoint. But not sure if this end point can be replaced with whisper endpoint? Please refer to the below link go to section 3.14 Service Endpoints. May be this can help you to suggest something. https://docs.unispeech.io/en/ums/ms-azure/speech-recog/usage

<service-endpoint enable="true" service-uri="http://10.0.0.1:5000/speech/recognition/interactive/cognitiveservices/v1"/>
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2024-01-12T05:44:24.0133333+00:00

@Shahnawaz This section is using an endpoint that points to speech to text container. Since this is using a local IP I think this product is using the speech service container to configure with your endpoint keys, this way your azure speech resource will get billed. But the whisper model is not available in a container as it is only available through batch endpoints of azure speech service hosted on Azure. I hope this helps!!
Shahnawaz 41 Reputation points

2024-01-12T07:49:47.3866667+00:00

Hello @Rohit Mungi , Thanks for sharing. Actually, my question is if instead of container endpoint can't we use batch endpoint. As you have mentioned, above there are regional, containerized and private endpoints. Also, if you refer to the below document they have suggested in section "5 Obtaining Service Credentials" on how to configure. Can you please take a look? We are getting no help from the third party. It will be a help if you can please help to demystify the implementation.
https://docs.unispeech.io/en/ums/ms-azure/speech-recog/install-rpm Adding Few key points from the documentation: In order to utilize either the deprecated Bing Speech API or the new Speech Service API, a corresponding service subscription key and an authentication endpoint need to be retrieved from the Microsoft Azure portal and further installed to the UniMRCP server.
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2024-01-12T10:58:47.38+00:00

@Shahnawaz Section 5.1 refers to creating a speech resource from Azure portal and then using the keys from the portal in this setup. From, the documentation I have read of unispeech it does not mention supporting batch endpoint. If you configure the endpoint to batch API I am not sure how the application setup changes in this case as batch endpoint does not give instant response, it rather provides a result URL that needs to be fetched at a later point to get the actual transcript. This is the same behavior that other AI services use with batch endpoints.
Shahnawaz 41 Reputation points

2024-01-15T01:30:25.2033333+00:00

Hello @Rohit Mungi , While reading the azure document I found the whisper model can be access via
"Whisper model via Azure AI Speech or via Azure OpenAI Service". I can read Azure AI speech is batch endpoint but Azure OpenAI Service to me does not seems to be batch endpoint. Any suggestion Azure OpenAI Service can help to achieve near to real time transcription of the recorded audio and is a good fit to uni-MRCP application. https://learn.microsoft.com/en-us/azure/ai-services/openai/whisper-quickstart?tabs=command-line&pivots=rest-api
Shahnawaz 41 Reputation points

2024-01-15T01:47:04.74+00:00

Hello @Rohit Mungi , While reading the Microsoft document I found that we can access whisper either via "Azure AI Speech or via Azure OpenAI Service". For Azure AI Speech it is the batch endpoint as you have mentioned previously but for Azure OpenAI Service from document it does not seems to be batch endpoint. Can get you suggestion on if Azure OpenAI Service can be used in Uni-MRCP endpoint and just to confirm this not the batch endpoint. Also, can we get transcription of audio steam near to real time?
https://learn.microsoft.com/en-us/azure/ai-services/openai/whisper-quickstart?tabs=command-line&pivots=rest-api
https://learn.microsoft.com/en-us/azure/ai-services/speech-service/whisper-overview

1 answer

Your answer

Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2024-01-11T11:30:34.0766667+00:00

@Shahnawaz I think in this case the installation and steps are related to 3rd party usage of Azure speech service, the documentation link above does not specify how this can be configured based on what I have read. This document though says that Azure speech can be configured for following endpoints. https://docs.unispeech.io/en/ums/ms-azure/speech-recog/install-rpm

The above does not specify if batch endpoint is supported, since whisper models are supported through batch transcription API, more details about Whisper model with Azure speech is documented here. I would recommend to also contact support of unispeech to confirm the same. Thanks!!
Shahnawaz 41 Reputation points

2024-01-12T00:58:04.6466667+00:00

Hello @Rohit Mungi , Thanks for the reply. While going through the document I found that in the configuration setting we can provide the endpoint. But not sure if this end point can be replaced with whisper endpoint? Please refer to the below link go to section 3.14 Service Endpoints. May be this can help you to suggest something. https://docs.unispeech.io/en/ums/ms-azure/speech-recog/usage

<service-endpoint enable="true" service-uri="http://10.0.0.1:5000/speech/recognition/interactive/cognitiveservices/v1"/>
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2024-01-12T05:44:24.0133333+00:00

@Shahnawaz This section is using an endpoint that points to speech to text container. Since this is using a local IP I think this product is using the speech service container to configure with your endpoint keys, this way your azure speech resource will get billed. But the whisper model is not available in a container as it is only available through batch endpoints of azure speech service hosted on Azure. I hope this helps!!
Shahnawaz 41 Reputation points

2024-01-12T07:49:47.3866667+00:00

Hello @Rohit Mungi , Thanks for sharing. Actually, my question is if instead of container endpoint can't we use batch endpoint. As you have mentioned, above there are regional, containerized and private endpoints. Also, if you refer to the below document they have suggested in section "5 Obtaining Service Credentials" on how to configure. Can you please take a look? We are getting no help from the third party. It will be a help if you can please help to demystify the implementation.
https://docs.unispeech.io/en/ums/ms-azure/speech-recog/install-rpm Adding Few key points from the documentation: In order to utilize either the deprecated Bing Speech API or the new Speech Service API, a corresponding service subscription key and an authentication endpoint need to be retrieved from the Microsoft Azure portal and further installed to the UniMRCP server.
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2024-01-12T10:58:47.38+00:00

@Shahnawaz Section 5.1 refers to creating a speech resource from Azure portal and then using the keys from the portal in this setup. From, the documentation I have read of unispeech it does not mention supporting batch endpoint. If you configure the endpoint to batch API I am not sure how the application setup changes in this case as batch endpoint does not give instant response, it rather provides a result URL that needs to be fetched at a later point to get the actual transcript. This is the same behavior that other AI services use with batch endpoints.
Shahnawaz 41 Reputation points

2024-01-15T01:30:25.2033333+00:00

Hello @Rohit Mungi , While reading the azure document I found the whisper model can be access via
"Whisper model via Azure AI Speech or via Azure OpenAI Service". I can read Azure AI speech is batch endpoint but Azure OpenAI Service to me does not seems to be batch endpoint. Any suggestion Azure OpenAI Service can help to achieve near to real time transcription of the recorded audio and is a good fit to uni-MRCP application. https://learn.microsoft.com/en-us/azure/ai-services/openai/whisper-quickstart?tabs=command-line&pivots=rest-api
Shahnawaz 41 Reputation points

2024-01-15T01:47:04.74+00:00

Hello @Rohit Mungi , While reading the Microsoft document I found that we can access whisper either via "Azure AI Speech or via Azure OpenAI Service". For Azure AI Speech it is the batch endpoint as you have mentioned previously but for Azure OpenAI Service from document it does not seems to be batch endpoint. Can get you suggestion on if Azure OpenAI Service can be used in Uni-MRCP endpoint and just to confirm this not the batch endpoint. Also, can we get transcription of audio steam near to real time?
https://learn.microsoft.com/en-us/azure/ai-services/openai/whisper-quickstart?tabs=command-line&pivots=rest-api
https://learn.microsoft.com/en-us/azure/ai-services/speech-service/whisper-overview

Answer 1

@Shahnawaz I am not sure about Uni-MRCP integration with Azure OpenAI, I would suggest to check with 3rd party support on this integration. With Respect to Azure OpenAI whisper deployment, you would first need access to Azure OpenAI which is mentioned as a pre-req in the document referenced. Once that is approved based on the criteria mentioned, you will be able to create an Azure OpenAI resource and then deploy an endpoint in a specific region which can be called using REST API. If REST API calls are allowed from Uni-MRCP you can setup the REST call to the Azure OpenAI resource. With Azure OpenAI the call is real time for small files upto 25MB. Anything beyond that, you will have to use speech service with batch endpoints. Thanks!!

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

How to integrate Avaya Aura Experience Portal (AAEP) and UniMRCP server running with Microsoft Azure Speech Recognition and Speech Synthesis

1 answer

Your answer