Edit

Get started with the Azure Communication Services Call Automation OpenAI sample

  • Article

The Azure Communication Services Call Automation OpenAI sample demonstrates how you can use Call Automation SDK and the recently announced public preview integration with Azure AI services to build intelligent virtual assistants.

In this sample, we'll cover off what this sample does and what you need as pre-requisites before we run this sample locally on your machine.

Download code

Find the project for this sample on GitHub. You can download this code and run it locally to try it for yourself.

Overview

This sample is a server-side application that helps you create a virtual assistant capable of handling calls using Call Automation and responding to customers using the newly announced integration with Azure AI services to provide AI capabilities such as Text-to-Speech and Speech-to-Text along with smart responses provided by Azure OpenAI.

This Azure Communication Services Call Automation AI sample demonstrates how to use the Call Automation SDK to answer an inbound call, recognizes user voice input using Call Automation recognize API with support for Speech-to-Text. Once input is recognized, it sends that information to OpenAI for an answer and plays the answer provided back by OpenAI to the caller using Call Automation play API with support for Text-to-Speech.

Prerequisites

Setup instructions

Before running this sample, you need to set up the resources mentioned in the 'Prerequisites' section with the following configuration updates:

1. Setup and host your Azure DevTunnel

Azure DevTunnels is an Azure service that enables you to share local web services hosted on the internet. Use the commands below to connect your local development environment to the public internet. This creates a tunnel with a persistent endpoint URL and which allows anonymous access. We'll then use this endpoint to notify your application of calling events from the Azure Communication Services Call Automation service.

devtunnel create --allow-anonymous
devtunnel port create -p 5165
devtunnel host

2. Add a Managed Identity to the Azure Communication Services Resource that connects to the Azure AI services resource

Follow the instructions in this documentation.

3. Add the required API Keys and endpoints

Open the appsettings.json file to configure the following settings:

  • DevTunnelUri: your dev tunnel endpoint
  • CognitiveServiceEndpoint: The Azure AI services endpoint
  • AcsConnectionString: Azure Communication Service resource's connection string.
  • AzureOpenAIServiceKey: Open AI's Service Key
  • AzureOpenAIServiceEndpoint: Open AI's Service Endpoint
  • AzureOpenAIDeploymentModelName: Open AI's Model name

Run the application

  1. Azure DevTunnel: Ensure your AzureDevTunnel URI is active and points to the correct port of your localhost application
  2. Run dotnet run to build and run the sample application
  3. Register an Event Grid Webhook for the IncomingCall Event that points to your DevTunnel URI. Instructions here.

Once that's completed, you should have a running application. The best way to test this sample is to place a call to your Azure Communication Services phone number and talk to your intelligent agent.

Next steps

Download code

Find the project for this sample on GitHub. You can download this code and run it locally to try it for yourself.

Overview

This sample is a server-side application that helps you create a virtual assistant capable of handling calls using Call Automation and responding to customers using the newly announced integration with Azure AI services to provide AI capabilities such as Text-to-Speech and Speech-to-Text along with smart responses provided by Azure OpenAI.

This Azure Communication Services Call Automation AI sample demonstrates how to use the Call Automation SDK to answer an inbound call. It recognizes user voice input using Call Automation recognize API with support for Speech-to-Text. Once input is recognized, it sends that information to OpenAI for an answer and plays the answer provided back by OpenAI to the caller using Call Automation play API with support for Text-to-Speech.

Prerequisites

Before running the sample for the first time

  • Open the application.yml file in the resources folder to configure the following settings

    • connectionstring: Azure Communication Service resource's connection string.
    • basecallbackuri: Base url of the app. For local development use dev tunnel url.
    • cognitiveServicesUrl: The Azure AI services endpoint
    • azureOpenAiServiceKey: Open AI's Service Key
    • azureOpenAiServiceEndpoint: Open AI's Service Endpoint
    • openAiModelName: Open AI's Model name

Setup and host your Azure DevTunnel

Azure DevTunnels is an Azure service that enables you to share local web services hosted on the internet. Use the commands provided to connect your local development environment to the public internet. This creates a tunnel with a persistent endpoint URL and which allows anonymous access. We'll then use this endpoint to notify your application of calling events from the Azure Communication Services Call Automation service.

devtunnel create --allow-anonymous
devtunnel port create -p 8080
devtunnel host

Run the application

  • Navigate to the directory containing the pom.xml file and use the following mvn commands:
    • Compile the application: mvn compile
    • Build the package: mvn package
    • Execute the app: mvn exec:java
  • Access the Swagger UI at http://localhost:8080/swagger-ui.html
    • Try the GET /outboundCall to run the Sample Application

Once that's completed, you should have a running application. The best way to test this sample is to place a call to your Azure Communication Services phone number and talk to your intelligent agent.

Next steps

Download code

Find the project for this sample on GitHub. You can download this code and run it locally to try it for yourself.

Overview

This sample is a server-side application that helps you create a virtual assistant capable of handling calls using Call Automation and responding to customers using the newly announced integration with Azure AI services to provide AI capabilities such as Text-to-Speech and Speech-to-Text along with smart responses provided by Azure OpenAI.

This Azure Communication Services Call Automation AI sample demonstrates how to use the Call Automation SDK to answer an inbound call, recognizes user voice input using Call Automation recognize API with support for Speech-to-Text. Once input is recognized, it sends that information to OpenAI for an answer and plays the answer that is provided back by OpenAI to the caller using Call Automation play API with support for Text-to-Speech.

Prerequisites

Setup instructions

Before running this sample, you need to set up the resources mentioned in the 'Prerequisites' section with the following configuration updates:

1. Setup and host your Azure DevTunnel

Azure DevTunnels is an Azure service that enables you to share local web services hosted on the internet. Use the commands provided in this document to connect your local development environment to the public internet. This creates a tunnel with a persistent endpoint URL and which allows anonymous access. We'll use this endpoint to notify your application of calling events from the ACS Call Automation service.

devtunnel create --allow-anonymous
devtunnel port create -p 8080
devtunnel host

2. Add a Managed Identity to the Azure Communication Services Resource that connects to the Azure AI services resource

Follow the instructions in this documentation.

3. Add the required API Keys and endpoints

Open the .env file to configure the following settings

  1. CONNECTION_STRING: Azure Communication Service resource's connection string.
  2. CALLBACK_URI: Base url of the app. (For local development replace the dev tunnel url)
  3. COGNITIVE_SERVICE_ENDPOINT: Azure AI Service endpoint
  4. AZURE_OPENAI_SERVICE_KEY: Azure OpenAI service key
  5. AZURE_OPENAI_SERVICE_ENDPOINT: Azure OpenAI endpoint
  6. AZURE_OPENAI_DEPLOYMENT_MODEL_NAME: Azure OpenAI deployment name
  7. AGENT_PHONE_NUMBER: Agent phone number to transfer the call to resolve queries

Run the application

  1. Open a new PowerShell window, cd into the callautomation-openai-sample folder and run npm run dev
  2. Browser should pop up with the below page. If not, navigate it to http://localhost:8080/
  3. Register an Event Grid Webhook for the IncomingCall Event that points to your DevTunnel URI. Instructions here.

Once that's completed, you should have a running application. The best way to test this sample is to place a call to your Azure Communication Services phone number and talk to your intelligent agent.

Next steps

Download code

Find the project for this sample on GitHub. You can download this code and run it locally to try it for yourself.

Overview

This sample is a server-side application that helps you create a virtual assistant capable of handling calls using Call Automation. It also enables your assistant to respond to customers using Azure AI services, which provide AI capabilities such as Text-to-Speech and Speech-to-Text, as well as smart responses provided by Azure OpenAI.

This Azure Communication Services Call Automation AI sample demonstrates how to use the Call Automation SDK to answer an inbound call, recognizes user voice input using Call Automation recognize API with support for Speech-to-Text. When the system recognizes the input, it sends the information to OpenAI for an answer and uses the Call Automation play API with support for Text-to-Speech to play back the answer provided by OpenAI to the caller.

Prerequisites

Setup instructions

Before running this sample, you need to set up the resources mentioned in the 'Prerequisites' section with the following configuration updates:

1. Setup Python environment

Create and activate python virtual environment and install required packages using following command

pip install -r requirements.txt
2. Setup and host your Azure DevTunnel

Azure DevTunnels is an Azure service that enables you to share local web services hosted on the internet. Use the commands provided to connect your local development environment to the public internet. This process creates a tunnel with a persistent endpoint URL and allows anonymous access. We'll use this endpoint to notify your application of calling events from the ACS Call Automation service.

devtunnel create --allow-anonymous
devtunnel port create -p 8080
devtunnel host

3. Add a Managed Identity to the Azure Communication Services Resource that connects to the Azure AI services resource

Follow the instructions in this documentation.

4. Add the required API Keys and endpoints

Open main.py file to configure the following settings

    • CALLBACK_URI_HOST: your dev tunnel endpoint
    • COGNITIVE_SERVICE_ENDPOINT: The Azure AI Services endpoint
    • ACS_CONNECTION_STRING: Azure Communication Service resource's connection string.
    • AZURE_OPENAI_SERVICE_KEY: Open AI's Service Key
    • AZURE_OPENAI_SERVICE_ENDPOINT: Open AI's Service Endpoint
    • AZURE_OPENAI_DEPLOYMENT_MODEL_NAME: Open AI's Model name
    • AGENT_PHONE_NUMBER: Agent Phone Number to transfer call

Run the application

  1. Navigate to callautomation-openai-sample folder and run main.py in debug mode or use command python ./main.py to run it from PowerShell, Command Prompt or Unix Terminal
  2. Browser should pop up with the below page. If not, navigate it to http://localhost:8080/ or your dev tunnel url.
  3. Register an Event Grid Webhook for the IncomingCall Event that points to your DevTunnel URI. Instructions here.

Once you complete that, your application should be up and running. The best way to test this sample is to place a call to your Azure Communication Services phone number and talk to your intelligent agent.

Next steps