Edit

Share via

Gather user input with Recognize action

  • Article

This guide helps you get started recognizing DTMF input provided by participants through Azure Communication Services Call Automation SDK.

Prerequisites

For AI features

Technical specifications

The following parameters are available to customize the Recognize function:

Parameter Type Default (if not specified) Description Required or Optional
Prompt

(For details, see Customize voice prompts to users with Play action)		 FileSource, TextSource Not set The message to play before recognizing input. Optional
InterToneTimeout TimeSpan 2 seconds

Min: 1 second
Max: 60 seconds		 Limit in seconds that Azure Communication Services waits for the caller to press another digit (inter-digit timeout). Optional
InitialSegmentationSilenceTimeoutInSeconds Integer 0.5 second How long recognize action waits for input before considering it a timeout. See How to recognize speech. Optional
RecognizeInputsType Enum dtmf Type of input that is recognized. Options are dtmf, choices, speech, and speechordtmf. Required
InitialSilenceTimeout TimeSpan 5 seconds

Min: 0 seconds
Max: 300 seconds (DTMF)
Max: 20 seconds (Choices)
Max: 20 seconds (Speech)		 Initial silence timeout adjusts how much nonspeech audio is allowed before a phrase before the recognition attempt ends in a "no match" result. See How to recognize speech. Optional
MaxTonesToCollect Integer No default

Min: 1		 Number of digits a developer expects as input from the participant. Required
StopTones IEnumeration<DtmfTone> Not set The digit participants can press to escape out of a batch DTMF event. Optional
InterruptPrompt Bool True If the participant has the ability to interrupt the playMessage by pressing a digit. Optional
InterruptCallMediaOperation Bool True If this flag is set, it interrupts the current call media operation. For example if any audio is being played it interrupts that operation and initiates recognize. Optional
OperationContext String Not set String that developers can pass mid action, useful for allowing developers to store context about the events they receive. Optional
Phrases String Not set List of phrases that associate to the label. Hearing any of these phrases results in a successful recognition. Required
Tone String Not set The tone to recognize if user decides to press a number instead of using speech. Optional
Label String Not set The key value for recognition. Required
Language String En-us The language that is used for recognizing speech. Optional
EndSilenceTimeout TimeSpan 0.5 second The final pause of the speaker used to detect the final result that gets generated as speech. Optional

Note

In situations where both DTMF and speech are in the recognizeInputsType, the recognize action acts on the first input type received. For example, if the user presses a keypad number first then the recognize action considers it a DTMF event and continues listening for DTMF tones. If the user speaks first then the recognize action considers it a speech recognition event and listens for voice input.

Create a new C# application

In the console window of your operating system, use the dotnet command to create a new web application.

dotnet new web -n MyApplication

Install the NuGet package

Get the NuGet package from NuGet Gallery | Azure.Communication.CallAutomation. Follow the instructions to install the package.

Establish a call

By this point you should be familiar with starting calls. For more information about making a call, see Quickstart: Make and outbound call. You can also use the code snippet provided here to understand how to answer a call.

var callAutomationClient = new CallAutomationClient("<Azure Communication Services connection string>");

var answerCallOptions = new AnswerCallOptions("<Incoming call context once call is connected>", new Uri("<https://sample-callback-uri>"))  
{  
    CallIntelligenceOptions = new CallIntelligenceOptions() { CognitiveServicesEndpoint = new Uri("<Azure Cognitive Services Endpoint>") } 
};  

var answerCallResult = await callAutomationClient.AnswerCallAsync(answerCallOptions);

Call the recognize action

When your application answers the call, you can provide information about recognizing participant input and playing a prompt.

DTMF

var maxTonesToCollect = 3;
String textToPlay = "Welcome to Contoso, please enter 3 DTMF.";
var playSource = new TextSource(textToPlay, "en-US-ElizabethNeural");
var recognizeOptions = new CallMediaRecognizeDtmfOptions(targetParticipant, maxTonesToCollect) {
  InitialSilenceTimeout = TimeSpan.FromSeconds(30),
    Prompt = playSource,
    InterToneTimeout = TimeSpan.FromSeconds(5),
    InterruptPrompt = true,
    StopTones = new DtmfTone[] {
      DtmfTone.Pound
    },
};
var recognizeResult = await callAutomationClient.GetCallConnection(callConnectionId)
  .GetCallMedia()
  .StartRecognizingAsync(recognizeOptions);

For speech-to-text flows, the Call Automation Recognize action also supports the use of custom speech models. Features like custom speech models can be useful when you're building an application that needs to listen for complex words that the default speech-to-text models may not understand. One example is when you're building an application for the telemedical industry and your virtual agent needs to be able to recognize medical terms. You can learn more in Create a custom speech project.

Speech-to-Text Choices

var choices = new List < RecognitionChoice > {
  new RecognitionChoice("Confirm", new List < string > {
    "Confirm",
    "First",
    "One"
  }) {
    Tone = DtmfTone.One
  },
  new RecognitionChoice("Cancel", new List < string > {
    "Cancel",
    "Second",
    "Two"
  }) {
    Tone = DtmfTone.Two
  }
};
String textToPlay = "Hello, This is a reminder for your appointment at 2 PM, Say Confirm to confirm your appointment or Cancel to cancel the appointment. Thank you!";

var playSource = new TextSource(textToPlay, "en-US-ElizabethNeural");
var recognizeOptions = new CallMediaRecognizeChoiceOptions(targetParticipant, choices) {
  InterruptPrompt = true,
    InitialSilenceTimeout = TimeSpan.FromSeconds(30),
    Prompt = playSource,
    OperationContext = "AppointmentReminderMenu",
    //Only add the SpeechModelEndpointId if you have a custom speech model you would like to use
    SpeechModelEndpointId = "YourCustomSpeechModelEndpointId"
};
var recognizeResult = await callAutomationClient.GetCallConnection(callConnectionId)
  .GetCallMedia()
  .StartRecognizingAsync(recognizeOptions);

Speech-to-Text

String textToPlay = "Hi, how can I help you today?";
var playSource = new TextSource(textToPlay, "en-US-ElizabethNeural");
var recognizeOptions = new CallMediaRecognizeSpeechOptions(targetParticipant) {
  Prompt = playSource,
    EndSilenceTimeout = TimeSpan.FromMilliseconds(1000),
    OperationContext = "OpenQuestionSpeech",
    //Only add the SpeechModelEndpointId if you have a custom speech model you would like to use
    SpeechModelEndpointId = "YourCustomSpeechModelEndpointId"
};
var recognizeResult = await callAutomationClient.GetCallConnection(callConnectionId)
  .GetCallMedia()
  .StartRecognizingAsync(recognizeOptions);

Speech-to-Text or DTMF

var maxTonesToCollect = 1; 
String textToPlay = "Hi, how can I help you today, you can press 0 to speak to an agent?"; 
var playSource = new TextSource(textToPlay, "en-US-ElizabethNeural"); 
var recognizeOptions = new CallMediaRecognizeSpeechOrDtmfOptions(targetParticipant, maxTonesToCollect) 
{ 
    Prompt = playSource, 
    EndSilenceTimeout = TimeSpan.FromMilliseconds(1000), 
    InitialSilenceTimeout = TimeSpan.FromSeconds(30), 
    InterruptPrompt = true, 
    OperationContext = "OpenQuestionSpeechOrDtmf",
    //Only add the SpeechModelEndpointId if you have a custom speech model you would like to use
    SpeechModelEndpointId = "YourCustomSpeechModelEndpointId" 
}; 
var recognizeResult = await callAutomationClient.GetCallConnection(callConnectionId) 
    .GetCallMedia() 
    .StartRecognizingAsync(recognizeOptions);

Note

If parameters aren't set, the defaults are applied where possible.

Receiving recognize event updates

Developers can subscribe to RecognizeCompleted and RecognizeFailed events on the registered webhook callback. Use this callback with business logic in your application to determine next steps when one of the events occurs.

Example of how you can deserialize the RecognizeCompleted event:

if (acsEvent is RecognizeCompleted recognizeCompleted) 
{ 
    switch (recognizeCompleted.RecognizeResult) 
    { 
        case DtmfResult dtmfResult: 
            //Take action for Recognition through DTMF 
            var tones = dtmfResult.Tones; 
            logger.LogInformation("Recognize completed succesfully, tones={tones}", tones); 
            break; 
        case ChoiceResult choiceResult: 
            // Take action for Recognition through Choices 
            var labelDetected = choiceResult.Label; 
            var phraseDetected = choiceResult.RecognizedPhrase; 
            // If choice is detected by phrase, choiceResult.RecognizedPhrase will have the phrase detected, 
            // If choice is detected using dtmf tone, phrase will be null 
            logger.LogInformation("Recognize completed succesfully, labelDetected={labelDetected}, phraseDetected={phraseDetected}", labelDetected, phraseDetected);
            break; 
        case SpeechResult speechResult: 
            // Take action for Recognition through Choices 
            var text = speechResult.Speech; 
            logger.LogInformation("Recognize completed succesfully, text={text}", text); 
            break; 
        default: 
            logger.LogInformation("Recognize completed succesfully, recognizeResult={recognizeResult}", recognizeCompleted.RecognizeResult); 
            break; 
    } 
}

Example of how you can deserialize the RecognizeFailed event:

if (acsEvent is RecognizeFailed recognizeFailed) 
{ 
    if (MediaEventReasonCode.RecognizeInitialSilenceTimedOut.Equals(recognizeFailed.ReasonCode)) 
    { 
        // Take action for time out 
        logger.LogInformation("Recognition failed: initial silencev time out"); 
    } 
    else if (MediaEventReasonCode.RecognizeSpeechOptionNotMatched.Equals(recognizeFailed.ReasonCode)) 
    { 
        // Take action for option not matched 
        logger.LogInformation("Recognition failed: speech option not matched"); 
    } 
    else if (MediaEventReasonCode.RecognizeIncorrectToneDetected.Equals(recognizeFailed.ReasonCode)) 
    { 
        // Take action for incorrect tone 
        logger.LogInformation("Recognition failed: incorrect tone detected"); 
    } 
    else 
    { 
        logger.LogInformation("Recognition failed, result={result}, context={context}", recognizeFailed.ResultInformation?.Message, recognizeFailed.OperationContext); 
    } 
}

Example of how you can deserialize the RecognizeCanceled event:

if (acsEvent is RecognizeCanceled { OperationContext: "AppointmentReminderMenu" })
        {
            logger.LogInformation($"RecognizeCanceled event received for call connection id: {@event.CallConnectionId}");
            //Take action on recognize canceled operation
           await callConnection.HangUpAsync(forEveryone: true);
        }

Prerequisites

For AI features

Technical specifications

The following parameters are available to customize the Recognize function:

Parameter Type Default (if not specified) Description Required or Optional
Prompt

(For details, see Customize voice prompts to users with Play action)		 FileSource, TextSource Not set The message to play before recognizing input. Optional
InterToneTimeout TimeSpan 2 seconds

Min: 1 second
Max: 60 seconds		 Limit in seconds that Azure Communication Services waits for the caller to press another digit (inter-digit timeout). Optional
InitialSegmentationSilenceTimeoutInSeconds Integer 0.5 second How long recognize action waits for input before considering it a timeout. See How to recognize speech. Optional
RecognizeInputsType Enum dtmf Type of input that is recognized. Options are dtmf, choices, speech, and speechordtmf. Required
InitialSilenceTimeout TimeSpan 5 seconds

Min: 0 seconds
Max: 300 seconds (DTMF)
Max: 20 seconds (Choices)
Max: 20 seconds (Speech)		 Initial silence timeout adjusts how much nonspeech audio is allowed before a phrase before the recognition attempt ends in a "no match" result. See How to recognize speech. Optional
MaxTonesToCollect Integer No default

Min: 1		 Number of digits a developer expects as input from the participant. Required
StopTones IEnumeration<DtmfTone> Not set The digit participants can press to escape out of a batch DTMF event. Optional
InterruptPrompt Bool True If the participant has the ability to interrupt the playMessage by pressing a digit. Optional
InterruptCallMediaOperation Bool True If this flag is set, it interrupts the current call media operation. For example if any audio is being played it interrupts that operation and initiates recognize. Optional
OperationContext String Not set String that developers can pass mid action, useful for allowing developers to store context about the events they receive. Optional
Phrases String Not set List of phrases that associate to the label. Hearing any of these phrases results in a successful recognition. Required
Tone String Not set The tone to recognize if user decides to press a number instead of using speech. Optional
Label String Not set The key value for recognition. Required
Language String En-us The language that is used for recognizing speech. Optional
EndSilenceTimeout TimeSpan 0.5 second The final pause of the speaker used to detect the final result that gets generated as speech. Optional

Note

In situations where both DTMF and speech are in the recognizeInputsType, the recognize action acts on the first input type received. For example, if the user presses a keypad number first then the recognize action considers it a DTMF event and continues listening for DTMF tones. If the user speaks first then the recognize action considers it a speech recognition event and listens for voice input.

Create a new Java application

In your terminal or command window, navigate to the directory where you would like to create your Java application. Run the mvn command to generate the Java project from the maven-archetype-quickstart template.

mvn archetype:generate -DgroupId=com.communication.quickstart -DartifactId=communication-quickstart -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

The mvn command creates a directory with the same name as the artifactId argument. The src/main/java directory contains the project source code. The src/test/java directory contains the test source.

Notice that the generate step created a directory with the same name as the artifactId. The src/main/java directory contains source code. The src/test/java directory contains tests. The pom.xml file is the project's Project Object Model (POM).

Update your applications POM file to use Java 8 or higher.

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
</properties>

Add package references

In your POM file, add the following reference for the project:

azure-communication-callautomation

<dependency>
  <groupId>com.azure</groupId>
  <artifactId>azure-communication-callautomation</artifactId>
  <version>1.0.0</version>
</dependency>

Establish a call

By this point you should be familiar with starting calls. For more information about making a call, see Quickstart: Make and outbound call. You can also use the code snippet provided here to understand how to answer a call.

CallIntelligenceOptions callIntelligenceOptions = new CallIntelligenceOptions().setCognitiveServicesEndpoint("https://sample-cognitive-service-resource.cognitiveservices.azure.com/"); 
answerCallOptions = new AnswerCallOptions("<Incoming call context>", "<https://sample-callback-uri>").setCallIntelligenceOptions(callIntelligenceOptions); 
Response < AnswerCallResult > answerCallResult = callAutomationClient
  .answerCallWithResponse(answerCallOptions)
  .block();

Call the recognize action

When your application answers the call, you can provide information about recognizing participant input and playing a prompt.

DTMF

var maxTonesToCollect = 3;
String textToPlay = "Welcome to Contoso, please enter 3 DTMF.";
var playSource = new TextSource() 
    .setText(textToPlay) 
    .setVoiceName("en-US-ElizabethNeural");

var recognizeOptions = new CallMediaRecognizeDtmfOptions(targetParticipant, maxTonesToCollect) 
    .setInitialSilenceTimeout(Duration.ofSeconds(30)) 
    .setPlayPrompt(playSource) 
    .setInterToneTimeout(Duration.ofSeconds(5)) 
    .setInterruptPrompt(true) 
    .setStopTones(Arrays.asList(DtmfTone.POUND));

var recognizeResponse = callAutomationClient.getCallConnectionAsync(callConnectionId) 
    .getCallMediaAsync() 
    .startRecognizingWithResponse(recognizeOptions) 
    .block(); 

log.info("Start recognizing result: " + recognizeResponse.getStatusCode());

For speech-to-text flows, the Call Automation Recognize action also supports the use of custom speech models. Features like custom speech models can be useful when you're building an application that needs to listen for complex words that the default speech-to-text models may not understand. One example is when you're building an application for the telemedical industry and your virtual agent needs to be able to recognize medical terms. You can learn more in Create a custom speech project.

Speech-to-Text Choices

var choices = Arrays.asList(
  new RecognitionChoice()
  .setLabel("Confirm")
  .setPhrases(Arrays.asList("Confirm", "First", "One"))
  .setTone(DtmfTone.ONE),
  new RecognitionChoice()
  .setLabel("Cancel")
  .setPhrases(Arrays.asList("Cancel", "Second", "Two"))
  .setTone(DtmfTone.TWO)
);

String textToPlay = "Hello, This is a reminder for your appointment at 2 PM, Say Confirm to confirm your appointment or Cancel to cancel the appointment. Thank you!";
var playSource = new TextSource()
  .setText(textToPlay)
  .setVoiceName("en-US-ElizabethNeural");
var recognizeOptions = new CallMediaRecognizeChoiceOptions(targetParticipant, choices)
  .setInterruptPrompt(true)
  .setInitialSilenceTimeout(Duration.ofSeconds(30))
  .setPlayPrompt(playSource)
  .setOperationContext("AppointmentReminderMenu")
  //Only add the SpeechRecognitionModelEndpointId if you have a custom speech model you would like to use
  .setSpeechRecognitionModelEndpointId("YourCustomSpeechModelEndpointID"); 
var recognizeResponse = callAutomationClient.getCallConnectionAsync(callConnectionId)
  .getCallMediaAsync()
  .startRecognizingWithResponse(recognizeOptions)
  .block();

Speech-to-Text

String textToPlay = "Hi, how can I help you today?"; 
var playSource = new TextSource() 
    .setText(textToPlay) 
    .setVoiceName("en-US-ElizabethNeural"); 
var recognizeOptions = new CallMediaRecognizeSpeechOptions(targetParticipant, Duration.ofMillis(1000)) 
    .setPlayPrompt(playSource) 
    .setOperationContext("OpenQuestionSpeech")
    //Only add the SpeechRecognitionModelEndpointId if you have a custom speech model you would like to use
    .setSpeechRecognitionModelEndpointId("YourCustomSpeechModelEndpointID");  
var recognizeResponse = callAutomationClient.getCallConnectionAsync(callConnectionId) 
    .getCallMediaAsync() 
    .startRecognizingWithResponse(recognizeOptions) 
    .block();

Speech-to-Text or DTMF

var maxTonesToCollect = 1; 
String textToPlay = "Hi, how can I help you today, you can press 0 to speak to an agent?"; 
var playSource = new TextSource() 
    .setText(textToPlay) 
    .setVoiceName("en-US-ElizabethNeural"); 
var recognizeOptions = new CallMediaRecognizeSpeechOrDtmfOptions(targetParticipant, maxTonesToCollect, Duration.ofMillis(1000)) 
    .setPlayPrompt(playSource) 
    .setInitialSilenceTimeout(Duration.ofSeconds(30)) 
    .setInterruptPrompt(true) 
    .setOperationContext("OpenQuestionSpeechOrDtmf")
    //Only add the SpeechRecognitionModelEndpointId if you have a custom speech model you would like to use
    .setSpeechRecognitionModelEndpointId("YourCustomSpeechModelEndpointID");  
var recognizeResponse = callAutomationClient.getCallConnectionAsync(callConnectionId) 
    .getCallMediaAsync() 
    .startRecognizingWithResponse(recognizeOptions) 
    .block();

Note

If parameters aren't set, the defaults are applied where possible.

Receiving recognize event updates

Developers can subscribe to RecognizeCompleted and RecognizeFailed events on the registered webhook callback. Use this callback with business logic in your application to determine next steps when one of the events occurs.

Example of how you can deserialize the RecognizeCompleted event:

if (acsEvent instanceof RecognizeCompleted) { 
    RecognizeCompleted event = (RecognizeCompleted) acsEvent; 
    RecognizeResult recognizeResult = event.getRecognizeResult().get(); 
    if (recognizeResult instanceof DtmfResult) { 
        // Take action on collect tones 
        DtmfResult dtmfResult = (DtmfResult) recognizeResult; 
        List<DtmfTone> tones = dtmfResult.getTones(); 
        log.info("Recognition completed, tones=" + tones + ", context=" + event.getOperationContext()); 
    } else if (recognizeResult instanceof ChoiceResult) { 
        ChoiceResult collectChoiceResult = (ChoiceResult) recognizeResult; 
        String labelDetected = collectChoiceResult.getLabel(); 
        String phraseDetected = collectChoiceResult.getRecognizedPhrase(); 
        log.info("Recognition completed, labelDetected=" + labelDetected + ", phraseDetected=" + phraseDetected + ", context=" + event.getOperationContext()); 
    } else if (recognizeResult instanceof SpeechResult) { 
        SpeechResult speechResult = (SpeechResult) recognizeResult; 
        String text = speechResult.getSpeech(); 
        log.info("Recognition completed, text=" + text + ", context=" + event.getOperationContext()); 
    } else { 
        log.info("Recognition completed, result=" + recognizeResult + ", context=" + event.getOperationContext()); 
    } 
}

Example of how you can deserialize the RecognizeFailed event:

if (acsEvent instanceof RecognizeFailed) { 
    RecognizeFailed event = (RecognizeFailed) acsEvent; 
    if (ReasonCode.Recognize.INITIAL_SILENCE_TIMEOUT.equals(event.getReasonCode())) { 
        // Take action for time out 
        log.info("Recognition failed: initial silence time out"); 
    } else if (ReasonCode.Recognize.SPEECH_OPTION_NOT_MATCHED.equals(event.getReasonCode())) { 
        // Take action for option not matched 
        log.info("Recognition failed: speech option not matched"); 
    } else if (ReasonCode.Recognize.DMTF_OPTION_MATCHED.equals(event.getReasonCode())) { 
        // Take action for incorrect tone 
        log.info("Recognition failed: incorrect tone detected"); 
    } else { 
        log.info("Recognition failed, result=" + event.getResultInformation().getMessage() + ", context=" + event.getOperationContext()); 
    } 
}

Example of how you can deserialize the RecognizeCanceled event:

if (acsEvent instanceof RecognizeCanceled) { 
    RecognizeCanceled event = (RecognizeCanceled) acsEvent; 
    log.info("Recognition canceled, context=" + event.getOperationContext()); 
}

Prerequisites

For AI features

Technical specifications

The following parameters are available to customize the Recognize function:

Parameter Type Default (if not specified) Description Required or Optional
Prompt

(For details, see Customize voice prompts to users with Play action)		 FileSource, TextSource Not set The message to play before recognizing input. Optional
InterToneTimeout TimeSpan 2 seconds

Min: 1 second
Max: 60 seconds		 Limit in seconds that Azure Communication Services waits for the caller to press another digit (inter-digit timeout). Optional
InitialSegmentationSilenceTimeoutInSeconds Integer 0.5 second How long recognize action waits for input before considering it a timeout. See How to recognize speech. Optional
RecognizeInputsType Enum dtmf Type of input that is recognized. Options are dtmf, choices, speech, and speechordtmf. Required
InitialSilenceTimeout TimeSpan 5 seconds

Min: 0 seconds
Max: 300 seconds (DTMF)
Max: 20 seconds (Choices)
Max: 20 seconds (Speech)		 Initial silence timeout adjusts how much nonspeech audio is allowed before a phrase before the recognition attempt ends in a "no match" result. See How to recognize speech. Optional
MaxTonesToCollect Integer No default

Min: 1		 Number of digits a developer expects as input from the participant. Required
StopTones IEnumeration<DtmfTone> Not set The digit participants can press to escape out of a batch DTMF event. Optional
InterruptPrompt Bool True If the participant has the ability to interrupt the playMessage by pressing a digit. Optional
InterruptCallMediaOperation Bool True If this flag is set, it interrupts the current call media operation. For example if any audio is being played it interrupts that operation and initiates recognize. Optional
OperationContext String Not set String that developers can pass mid action, useful for allowing developers to store context about the events they receive. Optional
Phrases String Not set List of phrases that associate to the label. Hearing any of these phrases results in a successful recognition. Required
Tone String Not set The tone to recognize if user decides to press a number instead of using speech. Optional
Label String Not set The key value for recognition. Required
Language String En-us The language that is used for recognizing speech. Optional
EndSilenceTimeout TimeSpan 0.5 second The final pause of the speaker used to detect the final result that gets generated as speech. Optional

Note

In situations where both DTMF and speech are in the recognizeInputsType, the recognize action acts on the first input type received. For example, if the user presses a keypad number first then the recognize action considers it a DTMF event and continues listening for DTMF tones. If the user speaks first then the recognize action considers it a speech recognition event and listens for voice input.

Create a new JavaScript application

Create a new JavaScript application in your project directory. Initialize a new Node.js project with the following command. This creates a package.json file for your project, which manages your project's dependencies.

npm init -y

Install the Azure Communication Services Call Automation package

npm install @azure/communication-call-automation

Create a new JavaScript file in your project directory, for example, name it app.js. Write your JavaScript code in this file.

Run your application using Node.js with the following command.

node app.js

Establish a call

By this point you should be familiar with starting calls. For more information about making a call, see Quickstart: Make and outbound call.

Call the recognize action

When your application answers the call, you can provide information about recognizing participant input and playing a prompt.

DTMF

const maxTonesToCollect = 3; 
const textToPlay = "Welcome to Contoso, please enter 3 DTMF."; 
const playSource: TextSource = { text: textToPlay, voiceName: "en-US-ElizabethNeural", kind: "textSource" }; 
const recognizeOptions: CallMediaRecognizeDtmfOptions = { 
    maxTonesToCollect: maxTonesToCollect, 
    initialSilenceTimeoutInSeconds: 30, 
    playPrompt: playSource, 
    interToneTimeoutInSeconds: 5, 
    interruptPrompt: true, 
    stopDtmfTones: [ DtmfTone.Pound ], 
    kind: "callMediaRecognizeDtmfOptions" 
}; 

await callAutomationClient.getCallConnection(callConnectionId) 
    .getCallMedia() 
    .startRecognizing(targetParticipant, recognizeOptions);

For speech-to-text flows, the Call Automation Recognize action also supports the use of custom speech models. Features like custom speech models can be useful when you're building an application that needs to listen for complex words that the default speech-to-text models may not understand. One example is when you're building an application for the telemedical industry and your virtual agent needs to be able to recognize medical terms. You can learn more in Create a custom speech project.

Speech-to-Text Choices

const choices = [ 
    {  
        label: "Confirm", 
        phrases: [ "Confirm", "First", "One" ], 
        tone: DtmfTone.One 
    }, 
    { 
        label: "Cancel", 
        phrases: [ "Cancel", "Second", "Two" ], 
        tone: DtmfTone.Two 
    } 
]; 

const textToPlay = "Hello, This is a reminder for your appointment at 2 PM, Say Confirm to confirm your appointment or Cancel to cancel the appointment. Thank you!"; 
const playSource: TextSource = { text: textToPlay, voiceName: "en-US-ElizabethNeural", kind: "textSource" }; 
const recognizeOptions: CallMediaRecognizeChoiceOptions = { 
    choices: choices, 
    interruptPrompt: true, 
    initialSilenceTimeoutInSeconds: 30, 
    playPrompt: playSource, 
    operationContext: "AppointmentReminderMenu", 
    kind: "callMediaRecognizeChoiceOptions",
    //Only add the speechRecognitionModelEndpointId if you have a custom speech model you would like to use
    speechRecognitionModelEndpointId: "YourCustomSpeechEndpointId"
}; 

await callAutomationClient.getCallConnection(callConnectionId) 
    .getCallMedia() 
    .startRecognizing(targetParticipant, recognizeOptions);

Speech-to-Text

const textToPlay = "Hi, how can I help you today?"; 
const playSource: TextSource = { text: textToPlay, voiceName: "en-US-ElizabethNeural", kind: "textSource" }; 
const recognizeOptions: CallMediaRecognizeSpeechOptions = { 
    endSilenceTimeoutInSeconds: 1, 
    playPrompt: playSource, 
    operationContext: "OpenQuestionSpeech", 
    kind: "callMediaRecognizeSpeechOptions",
    //Only add the speechRecognitionModelEndpointId if you have a custom speech model you would like to use
    speechRecognitionModelEndpointId: "YourCustomSpeechEndpointId"
}; 

await callAutomationClient.getCallConnection(callConnectionId) 
    .getCallMedia() 
    .startRecognizing(targetParticipant, recognizeOptions);

Speech-to-Text or DTMF

const maxTonesToCollect = 1; 
const textToPlay = "Hi, how can I help you today, you can press 0 to speak to an agent?"; 
const playSource: TextSource = { text: textToPlay, voiceName: "en-US-ElizabethNeural", kind: "textSource" }; 
const recognizeOptions: CallMediaRecognizeSpeechOrDtmfOptions = { 
    maxTonesToCollect: maxTonesToCollect, 
    endSilenceTimeoutInSeconds: 1, 
    playPrompt: playSource, 
    initialSilenceTimeoutInSeconds: 30, 
    interruptPrompt: true, 
    operationContext: "OpenQuestionSpeechOrDtmf", 
    kind: "callMediaRecognizeSpeechOrDtmfOptions",
    //Only add the speechRecognitionModelEndpointId if you have a custom speech model you would like to use
    speechRecognitionModelEndpointId: "YourCustomSpeechEndpointId"
}; 

await callAutomationClient.getCallConnection(callConnectionId) 
    .getCallMedia() 
    .startRecognizing(targetParticipant, recognizeOptions);

Note

If parameters aren't set, the defaults are applied where possible.

Receiving recognize event updates

Developers can subscribe to RecognizeCompleted and RecognizeFailed events on the registered webhook callback. Use this callback with business logic in your application to determine next steps when one of the events occurs.

Example of how you can deserialize the RecognizeCompleted event:

if (event.type === "Microsoft.Communication.RecognizeCompleted") { 
    if (eventData.recognitionType === "dtmf") { 
        const tones = eventData.dtmfResult.tones; 
        console.log("Recognition completed, tones=%s, context=%s", tones, eventData.operationContext); 
    } else if (eventData.recognitionType === "choices") { 
        const labelDetected = eventData.choiceResult.label; 
        const phraseDetected = eventData.choiceResult.recognizedPhrase; 
        console.log("Recognition completed, labelDetected=%s, phraseDetected=%s, context=%s", labelDetected, phraseDetected, eventData.operationContext); 
    } else if (eventData.recognitionType === "speech") { 
        const text = eventData.speechResult.speech; 
        console.log("Recognition completed, text=%s, context=%s", text, eventData.operationContext); 
    } else { 
        console.log("Recognition completed: data=%s", JSON.stringify(eventData, null, 2)); 
    } 
}

Example of how you can deserialize the RecognizeFailed event:

if (event.type === "Microsoft.Communication.RecognizeFailed") {
    console.log("Recognize failed: data=%s", JSON.stringify(eventData, null, 2));
}

Example of how you can deserialize the RecognizeCanceled event:

if (event.type === "Microsoft.Communication.RecognizeCanceled") {
    console.log("Recognize canceled, context=%s", eventData.operationContext);
}

Prerequisites

For AI features

Technical specifications

The following parameters are available to customize the Recognize function:

Parameter Type Default (if not specified) Description Required or Optional
Prompt

(For details, see Customize voice prompts to users with Play action)		 FileSource, TextSource Not set The message to play before recognizing input. Optional
InterToneTimeout TimeSpan 2 seconds

Min: 1 second
Max: 60 seconds		 Limit in seconds that Azure Communication Services waits for the caller to press another digit (inter-digit timeout). Optional
InitialSegmentationSilenceTimeoutInSeconds Integer 0.5 second How long recognize action waits for input before considering it a timeout. See How to recognize speech. Optional
RecognizeInputsType Enum dtmf Type of input that is recognized. Options are dtmf, choices, speech, and speechordtmf. Required
InitialSilenceTimeout TimeSpan 5 seconds

Min: 0 seconds
Max: 300 seconds (DTMF)
Max: 20 seconds (Choices)
Max: 20 seconds (Speech)		 Initial silence timeout adjusts how much nonspeech audio is allowed before a phrase before the recognition attempt ends in a "no match" result. See How to recognize speech. Optional
MaxTonesToCollect Integer No default

Min: 1		 Number of digits a developer expects as input from the participant. Required
StopTones IEnumeration<DtmfTone> Not set The digit participants can press to escape out of a batch DTMF event. Optional
InterruptPrompt Bool True If the participant has the ability to interrupt the playMessage by pressing a digit. Optional
InterruptCallMediaOperation Bool True If this flag is set, it interrupts the current call media operation. For example if any audio is being played it interrupts that operation and initiates recognize. Optional
OperationContext String Not set String that developers can pass mid action, useful for allowing developers to store context about the events they receive. Optional
Phrases String Not set List of phrases that associate to the label. Hearing any of these phrases results in a successful recognition. Required
Tone String Not set The tone to recognize if user decides to press a number instead of using speech. Optional
Label String Not set The key value for recognition. Required
Language String En-us The language that is used for recognizing speech. Optional
EndSilenceTimeout TimeSpan 0.5 second The final pause of the speaker used to detect the final result that gets generated as speech. Optional

Note

In situations where both DTMF and speech are in the recognizeInputsType, the recognize action acts on the first input type received. For example, if the user presses a keypad number first then the recognize action considers it a DTMF event and continues listening for DTMF tones. If the user speaks first then the recognize action considers it a speech recognition event and listens for voice input.

Create a new Python application

Set up a Python virtual environment for your project

python -m venv play-audio-app

Activate your virtual environment

On Windows, use the following command:

.\ play-audio-quickstart \Scripts\activate

On Unix, use the following command:

source play-audio-quickstart /bin/activate

Install the Azure Communication Services Call Automation package

pip install azure-communication-callautomation

Create your application file in your project directory, for example, name it app.py. Write your Python code in this file.

Run your application using Python with the following command.

python app.py

Establish a call

By this point you should be familiar with starting calls. For more information about making a call, see Quickstart: Make and outbound call.

Call the recognize action

When your application answers the call, you can provide information about recognizing participant input and playing a prompt.

DTMF

max_tones_to_collect = 3 
text_to_play = "Welcome to Contoso, please enter 3 DTMF." 
play_source = TextSource(text=text_to_play, voice_name="en-US-ElizabethNeural") 
call_automation_client.get_call_connection(call_connection_id).start_recognizing_media( 
    dtmf_max_tones_to_collect=max_tones_to_collect, 
    input_type=RecognizeInputType.DTMF, 
    target_participant=target_participant, 
    initial_silence_timeout=30, 
    play_prompt=play_source, 
    dtmf_inter_tone_timeout=5, 
    interrupt_prompt=True, 
    dtmf_stop_tones=[ DtmfTone.Pound ])

For speech-to-text flows, the Call Automation Recognize action also supports the use of custom speech models. Features like custom speech models can be useful when you're building an application that needs to listen for complex words that the default speech-to-text models may not understand. One example is when you're building an application for the telemedical industry and your virtual agent needs to be able to recognize medical terms. You can learn more in Create a custom speech project.

Speech-to-Text Choices

choices = [ 
    RecognitionChoice( 
        label="Confirm", 
        phrases=[ "Confirm", "First", "One" ], 
        tone=DtmfTone.ONE 
    ), 
    RecognitionChoice( 
        label="Cancel", 
        phrases=[ "Cancel", "Second", "Two" ], 
        tone=DtmfTone.TWO 
    ) 
] 
text_to_play = "Hello, This is a reminder for your appointment at 2 PM, Say Confirm to confirm your appointment or Cancel to cancel the appointment. Thank you!" 
play_source = TextSource(text=text_to_play, voice_name="en-US-ElizabethNeural") 
call_automation_client.get_call_connection(call_connection_id).start_recognizing_media( 
    input_type=RecognizeInputType.CHOICES, 
    target_participant=target_participant, 
    choices=choices, 
    interrupt_prompt=True, 
    initial_silence_timeout=30, 
    play_prompt=play_source, 
    operation_context="AppointmentReminderMenu",
    # Only add the speech_recognition_model_endpoint_id if you have a custom speech model you would like to use
    speech_recognition_model_endpoint_id="YourCustomSpeechModelEndpointId")

Speech-to-Text

text_to_play = "Hi, how can I help you today?" 
play_source = TextSource(text=text_to_play, voice_name="en-US-ElizabethNeural") 
call_automation_client.get_call_connection(call_connection_id).start_recognizing_media( 
    input_type=RecognizeInputType.SPEECH, 
    target_participant=target_participant, 
    end_silence_timeout=1, 
    play_prompt=play_source, 
    operation_context="OpenQuestionSpeech",
    # Only add the speech_recognition_model_endpoint_id if you have a custom speech model you would like to use
    speech_recognition_model_endpoint_id="YourCustomSpeechModelEndpointId")

Speech-to-Text or DTMF

max_tones_to_collect = 1 
text_to_play = "Hi, how can I help you today, you can also press 0 to speak to an agent." 
play_source = TextSource(text=text_to_play, voice_name="en-US-ElizabethNeural") 
call_automation_client.get_call_connection(call_connection_id).start_recognizing_media( 
    dtmf_max_tones_to_collect=max_tones_to_collect, 
    input_type=RecognizeInputType.SPEECH_OR_DTMF, 
    target_participant=target_participant, 
    end_silence_timeout=1, 
    play_prompt=play_source, 
    initial_silence_timeout=30, 
    interrupt_prompt=True, 
    operation_context="OpenQuestionSpeechOrDtmf",
    # Only add the speech_recognition_model_endpoint_id if you have a custom speech model you would like to use
    speech_recognition_model_endpoint_id="YourCustomSpeechModelEndpointId")  
app.logger.info("Start recognizing")

Note

If parameters aren't set, the defaults are applied where possible.

Receiving recognize event updates

Developers can subscribe to RecognizeCompleted and RecognizeFailed events on the registered webhook callback. Use this callback with business logic in your application to determine next steps when one of the events occurs.

Example of how you can deserialize the RecognizeCompleted event:

if event.type == "Microsoft.Communication.RecognizeCompleted": 
    app.logger.info("Recognize completed: data=%s", event.data) 
    if event.data['recognitionType'] == "dtmf": 
        tones = event.data['dtmfResult']['tones'] 
        app.logger.info("Recognition completed, tones=%s, context=%s", tones, event.data.get('operationContext')) 
    elif event.data['recognitionType'] == "choices": 
        labelDetected = event.data['choiceResult']['label']; 
        phraseDetected = event.data['choiceResult']['recognizedPhrase']; 
        app.logger.info("Recognition completed, labelDetected=%s, phraseDetected=%s, context=%s", labelDetected, phraseDetected, event.data.get('operationContext')); 
    elif event.data['recognitionType'] == "speech": 
        text = event.data['speechResult']['speech']; 
        app.logger.info("Recognition completed, text=%s, context=%s", text, event.data.get('operationContext')); 
    else: 
        app.logger.info("Recognition completed: data=%s", event.data);

Example of how you can deserialize the RecognizeFailed event:

if event.type == "Microsoft.Communication.RecognizeFailed": 
    app.logger.info("Recognize failed: data=%s", event.data);

Example of how you can deserialize the RecognizeCanceled event:

if event.type == "Microsoft.Communication.RecognizeCanceled":
    # Handle the RecognizeCanceled event according to your application logic

Event codes

Status Code Subcode Message
RecognizeCompleted 200 8531 Action completed, max digits received.
RecognizeCompleted 200 8514 Action completed as stop tone was detected.
RecognizeCompleted 400 8508 Action failed, the operation was canceled.
RecognizeCompleted 400 8532 Action failed, inter-digit silence timeout reached.
RecognizeCanceled 400 8508 Action failed, the operation was canceled.
RecognizeFailed 400 8510 Action failed, initial silence timeout reached.
RecognizeFailed 500 8511 Action failed, encountered failure while trying to play the prompt.
RecognizeFailed 500 8512 Unknown internal server error.
RecognizeFailed 400 8510 Action failed, initial silence timeout reached
RecognizeFailed 400 8532 Action failed, inter-digit silence timeout reached.
RecognizeFailed 400 8565 Action failed, bad request to Azure AI services. Check input parameters.
RecognizeFailed 400 8565 Action failed, bad request to Azure AI services. Unable to process payload provided, check the play source input.
RecognizeFailed 401 8565 Action failed, Azure AI services authentication error.
RecognizeFailed 403 8565 Action failed, forbidden request to Azure AI services, free subscription used by the request ran out of quota.
RecognizeFailed 429 8565 Action failed, requests exceeded the number of allowed concurrent requests for the Azure AI services subscription.
RecognizeFailed 408 8565 Action failed, request to Azure AI services timed out.
RecognizeFailed 500 8511 Action failed, encountered failure while trying to play the prompt.
RecognizeFailed 500 8512 Unknown internal server error.

Known limitations

  • In-band DTMF isn't supported. Use RFC 2833 DTMF instead.
  • Text-to-Speech text prompts support a maximum of 400 characters, if your prompt is longer than this we suggest using SSML for Text-to-Speech based play actions.
  • For scenarios where you exceed your Speech service quota limit, you can request to increase this limit by following the steps outlined in Speech services quotas and limits.

Clean up resources

If you want to clean up and remove a Communication Services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it. Learn more about cleaning up resources.

Next Steps