How to link and use Azure Communication Services with Azure Cognitive Services (Text-to-speech) in C#?

Question

Hi, we're using Azure Communication Services (ACS) to send Emails, SMS text messages and make Voice phone calls (Robo calls). Everything is outbound communication. 

For Voice, we want to use the Text-to-speech (TTS) feature of Azure Cognitive Services.

Linking the ACS to the Cognitive service is where the issue is.

There are several examples by Microsoft itself. E.g.,

• Learn Call Automation with Text-to-speech (published 04/23/2023)
• YouTube video using CallAutomation and Text-to-speech (published 05/24/2023)

A prerequisite in that article is that ACS is linked to the Cognitive service, using following documentation: https://learn.microsoft.com/en-us/azure/communication-services/concepts/call-automation/azure-communication-services-azure-cognitive-services-integration

However, documentation only seems to be applicable to the Azure Portal in Private Preview. It mentions a "Settings > Identity" tab at the ACS, while this is not present. We want to avoid applying in the early adopter program for Private Preview. Maybe this gives additional options in the Azure Portal, but we want to deploy our solution to the customer soon, and we cannot expect the customer to enroll their Azure Portal for Private Preview as well.

Alternative

As this Settings > Identity tab is not visible, I tried to use the alternative from the accepted answer in this Q&A: https://learn.microsoft.com/en-us/answers/questions/1319535/azure-communication-service-can-not-see-identity-t

• Added an AAD with as scopes: the Id found under ACS > Settings > Properties > Id
• In the Cognitive Speech Service > Access control (IAM) > Added role assignment for the earlier created AAD (Roles Cognitive Services Speech User and Cognitive Services User)

To no avail so far, please see the code below.

Code

Azure SDK: I'm using an Azure Function App (C#) to use the Azure CallAutomation SDK. Currently using Azure.Communication.CallAutomation, version 1.1.0-alpha.20230720.1 from the Azure SDK's.

When the CallbackUri API is reached, I'm looping the CloudEvents. Whenever the CallConnected event is triggered, I use CallMedia.PlayToAllAsync() with TextSource as PlaySource parameter:

string textToPlay = "Welcome to Contoso";

TextSource playSource = new TextSource(textToPlay)
{
    VoiceName = "en-US-NancyNeural"
};

await callConnectionMedia.PlayToAllAsync(playSource);

This throws following error (upon PlayToAllAsync()):

Azure.Communication.CallAutomation: Request not allowed when Cognitive Service Configuration not set during call setup.
Status: 405 (Method Not Allowed)
ErrorCode: 8522

Content:
{"error":{"code":"8522","message":"Request not allowed when Cognitive Service Configuration not set during call setup."}}

Headers:
X-Microsoft-Skype-Chain-ID: REDACTED
x-ms-client-request-id: 854933a9-fe36-42a7-ae1e-c523e81a7555
X-Cache: REDACTED
X-Azure-Ref: REDACTED
Date: Tue, 25 Jul 2023 10:00:32 GMT
Content-Length: 121
Content-Type: application/json; charset=utf-8

I added the following line to code, when I instantiate the CallAutomationClient:

var createCallOption = new CreateCallOptions(callInvite, callbackUri);    

createCallOption.AzureCognitiveServicesEndpointUrl = new Uri("https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken");                          

try
{
	var response = await callAutomationClient.CreateCallAsync(createCallOption).ConfigureAwait(false);
    [...]

The CreateCallOptions.AzureCoginitiveServicesEndpointUrl is not even documented yet. Anyway, when trying this, I get following error upon calling CallAutomationClient.CreateCallAsync():

Service Principal for the Azure Communication Service Resource is not configured.
Status: 400 (Bad Request)
ErrorCode: 8567

Content:
"error\":{\"code\":\"8567\",\"message\":\"Service Principal for the Azure Communication Service Resource is not configured.\"}}\r\n\r\nHeaders:\r\nx-ms-client-request-id: 2aa74482-327e-429b-a78f-e8e942ecd1ee\r\nX-Microsoft-Skype-Chain-ID: REDACTED\r\nX-Cache: REDACTED\r\nX-Azure-Ref: REDACTED\r\nDate: Tue, 25 Jul 2023 10:13:09 GMT\r\nContent-Length: 119\r\nContent-Type: application/json; charset=utf-8\r\n

Q: How can I link ACS to the Cognitive service then?

Thanks in advance!

Answer 1

Hi Andrew,

After enrolling in the private preview program and being in direct contact with Microsoft, I got it working. I'm working with the now public preview code:

• NuGet: Azure.Communication.CallAutomation version 1.1.0-beta.1
• CallAutomationClientOptions.ServiceVersion.V2023_06_15_Preview

Important to know is that I needed a Multi Service Azure Cognitive Service instead of a single Speech Service, and then use the endpoint of that one when establishing the call. The endpoint of that Multi Service service now includes the name of it, instead of the general "<location>.api.cognitive.microsoft.com/sts/v1.0/issuetoken".

Note that the documentation on how to link ACS to an Azure Cognitive Service has been updated when going from private to public preview: https://learn.microsoft.com/en-us/azure/communication-services/concepts/call-automation/azure-communication-services-azure-cognitive-services-integration

Please see some of my code snippets below.

Initialization of the CallAutomationClient (Setting the CallAutomationClientOptions is optional, it also works when not passing it to the constructor):

new CallAutomationClient(currentConfig["ACSConnectionString"], BuildCallAutomationClientOptions())

protected CallAutomationClientOptions BuildCallAutomationClientOptions()
{
	CallAutomationClientOptions ret = new CallAutomationClientOptions();

	ret.AddPolicy(new Catch429Policy(), Azure.Core.HttpPipelinePosition.PerRetry);

	return ret;
}

Establishing the call:

var identifierFrom = new PhoneNumberIdentifier(Args.PhoneNumberFrom); 
var identifierTo = new PhoneNumberIdentifier(Args.PhoneNumberTo);  
var callInvite = new CallInvite(identifierTo, identifierFrom);

CreateCallOptions createCallOption = new(callInvite, callbackUri) 
{ 	
	CognitiveServicesEndpoint = new Uri(cognitiveServicesEndPointString) 
};

await callAutomationClient.CreateCallAsync(createCallOption);

Note: cognitive services endpoint string = https://<my-multi-service-account-cognitive-service-name>.cognitiveservices.azure.com/

Later, upon callback, when the call is connected and I want to use text-to-speech to prompt an audio message, I do this:

TextSource playSource = new TextSource("text to play in the audio message")
{
	VoiceName = "en-US-NancyNeural"
};

PlayToAllOptions playToAllOptions = new PlayToAllOptions(playSource)
{
	Loop = false
};

await callConnectionMedia.PlayToAllAsync(playToAllOptions);