QuickStart: Add closed captions to your calling app

Important

Functionality described in this article is currently in public preview. This preview version is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

• Azure account with an active subscription, for details see Create an account for free.
• Azure Communication Services resource. See Create an Azure Communication Services resource. Save the connection string for this resource.
• An app with voice and video calling, refer to our Voice and Video calling quickstarts.

Note

Please note that you will need to have a voice calling app using Azure Communication Services calling SDKs to access the closed captions feature that is described in this guide.

Models

Name	Description
CaptionsCallFeature	API for Captions
CaptionsCommon	Base class for captions
StartCaptionOptions	Closed caption options like spoken language
CaptionsHandler	Callback definition for handling CaptionsReceivedEventType event
CaptionsInfo	Data structure received for each CaptionsReceivedEventType event

Get closed captions feature

let captionsCallFeature: SDK.CaptionsCallFeature = call.feature(SDK.Features.Captions);

Get captions object

You need to get and cast the Captions object to utilize Captions specific features.

let captions: SDK.Captions;
if (captionsCallFeature.captions.kind === 'Captions') {
    captions = captionsCallFeature.captions as SDK.Captions;
}

Subscribe to listeners

Add a listener to receive captions active/inactive status

const captionsActiveChangedHandler = () => {
    if (captions.isCaptionsFeatureActive()) {
        /* USER CODE HERE - E.G. RENDER TO DOM */
    }
}
captions.on('CaptionsActiveChanged', captionsActiveChangedHandler);

Add a listener for captions data received

Handle the returned CaptionsInfo data object.

Note: The object contains a resultType prop that indicates whether the data is a partial caption or a finalized version of the caption. ResultType Partial indicates live unedited caption, while Final indicates a finalized interpreted version of the sentence (i.e includes punctuation and capitalization).

const captionsReceivedHandler : CaptionsHandler = (data: CaptionsInfo) => { 
    /** USER CODE HERE - E.G. RENDER TO DOM 
     * data.resultType
     * data.speaker
     * data.spokenLanguage
     * data.spokenText
     * data.timeStamp
    */
   // Example code:
   // Create a dom element, i.e. div, with id "captionArea" before proceeding with the sample code
    let mri: string;
    switch (data.speaker.identifier.kind) {
        case 'communicationUser': { mri = data.speaker.identifier.communicationUserId; break; }
        case 'phoneNumber': { mri = data.speaker.identifier.phoneNumber; break; }
    }
    const outgoingCaption = `prefix${mri.replace(/:/g, '').replace(/-/g, '')}`;

    let captionArea = document.getElementById("captionArea");
    const captionText = `${data.timestamp.toUTCString()}
        ${data.speaker.displayName}: ${data.spokenText}`;

    let foundCaptionContainer = captionArea.querySelector(`.${outgoingCaption}[isNotFinal='true']`);
    if (!foundCaptionContainer) {
        let captionContainer = document.createElement('div');
        captionContainer.setAttribute('isNotFinal', 'true');
        captionContainer.style['borderBottom'] = '1px solid';
        captionContainer.style['whiteSpace'] = 'pre-line';
        captionContainer.textContent = captionText;
        captionContainer.classList.add(newClassName);

        captionArea.appendChild(captionContainer);
    } else {
        foundCaptionContainer.textContent = captionText;

        if (captionData.resultType === 'Final') {
            foundCaptionContainer.setAttribute('isNotFinal', 'false');
        }
    }
}; 
captions.on('CaptionsReceived', captionsReceivedHandler); 

Add a listener to receive spoken language changed status

const spokenLanguageChangedHandler = () => {
    if (captions.activeSpokenLanguage !== currentSpokenLanguage) {
        /* USER CODE HERE - E.G. RENDER TO DOM */
    }
}
captions.on('SpokenLanguageChanged', spokenLanguageChangedHandler)

Start captions

Once you have set up all your listeners, you can now start adding captions.

try {
    await captions.startCaptions({ spokenLanguage: 'en-us' });
} catch (e) {
    /* USER ERROR HANDLING CODE HERE */
}

Stop captions

try {
    captions.stopCaptions(); 
} catch (e) {
    /* USER ERROR HANDLING CODE HERE */
}

Unsubscribe to listeners

captions.off('CaptionsActiveChanged', captionsActiveChangedHandler);
captions.off('CaptionsReceived', captionsReceivedHandler); 

Spoken language support

Get a list of supported spoken languages

Get a list of supported spoken languages that your users can select from when enabling closed captions. The property returns an array of languages in bcp 47 format.

const spokenLanguages = captions.supportedSpokenLanguages; 

Set spoken language

Pass a value in from the supported spoken languages array to ensure that the requested language is supported. By default, if contoso provides no language or an unsupported language, the spoken language defaults to 'en-us'.

// bcp 47 formatted language code
const language = 'en-us'; 

// Altneratively, pass a value from the supported spoken languages array
const language = spokenLanguages[0]; 

try {
    captions.setSpokenLanguage(language);
} catch (e) {
    /* USER ERROR HANDLING CODE HERE */
}

Prerequisites

Refer to the Voice Calling Quickstart to set up a sample app with voice calling.

Models

Name	Description
CaptionsCallFeature	Used to start and manage closed captions.
StartCaptionsOptions	Used for representing options to start closed captions.
CaptionsListener	Used to handle captions events.
CaptionsInfo	Used to representing captions data.

Methods

Start captions

1. Get the ongoing call object established during the prerequisite steps.
2. Get the Captions Feature object

CaptionsCallFeature captionsCallFeature = call.api(Features.CAPTIONS);

3. Define the CaptionsListener

CaptionsListener captionsListener = new CaptionsListener() {
    @Override
    public void onCaptions(CaptionsInfo captionsInfo) {
        String captionsText = captionsInfo.getText(); // get transcribed text
        String speakerName = captionsInfo.getSpeaker().getDisplayName; // get display name of current speaker
        Date timeStamp = captionsInfo.getTimestamp(); // get timestamp corresponding to caption

        // display captionsText and information as needed...
    }
};

4. Register the captionsListener
5. Invoke startCaptions with the desired options.

public void startCallCaptions() {
    captionsCallFeature.addOnCaptionsReceivedListener(captionsListener);
    if (!captionsCallFeature.isCaptionsActive) {
        StartCaptionsOptions startCaptionsOptions = new StartCaptionsOptions();
        startCaptionsOptions.setLanguage("en-us");

        try {
            captionsCallFeature.startCaptions(startCaptionsOptions);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Stopping captions

Remove the previously registered captionsListener.

public void stopCaptions() {
    captionsCallFeature.removeOnCaptionsReceivedListener(captionsListener);
}

Get available languages

Call the getAvailableLanguages method on the CaptionsCallFeature API.

String[] captionsAvailableLanguages = captionsCallFeature.getAvailableLanguages();

Update language

Pass a value in from the available languages array to ensure that the requested language is supported.

public void switchCaptionsLanguage() {
    if (!captionsCallFeature.isCaptionsActive) {
        return;
    }
    try {
        captionsCallFeature.select(captionsAvailableLanguages[0]);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Clean up

Learn more about cleaning up resources here.

Prerequisites

Refer to the Voice Calling Quickstart to set up a sample app with voice calling.

Models

Name	Description
CaptionsCallFeature	Used to start and manage closed captions.
StartCaptionsOptions	Used for representing options to start closed captions.
CaptionsCallFeatureDelegate	Used to handle captions events.
CaptionsInfo	Used to representing captions data.

Methods

Start captions

1. Get the ongoing call object established during the prerequisite steps.
2. Get the Captions Feature object

var captionsFeature = self.call!.feature(Features.captions)

3. Define the CaptionsCallFeatureDelegate

@State var callObserver:CallObserver?
extension CallObserver: CaptionsCallFeatureDelegate {
    init(view:<nameOfView>) {
        owner = view
        super.init()
    }
    public func captionsCallFeature(_ captionsFeature: CaptionsCallFeature, didChangeCaptionsState args: PropertyChangedEventArgs) {
        os_log("Call captions state changed to %d", log:log, captionsFeature.isCaptionsActive)
    }
    
    public func captionsCallFeature(_ captionsFeature: CaptionsCallFeature, didReceiveCaptions: CaptionsInfo) {
        let formatter = DateFormatter()
        formatter.dateStyle = .short
        formatter.timeStyle = .short
        let displayedTime = formatter.string(from: didReceiveCaptions.timestamp)
        let captionsText = "\(displayedTime) \(didReceiveCaptions.speaker.displayName): \(didReceiveCaptions.text)"
        // next display captionsText
    }
}
self.callObserver = CallObserver(view:self)

4. Register the captions feature delegate.
5. Invoke startCaptions with the desired options.

func startCaptions() {
    captionsFeature!.delegate = self.callObserver
    if(!captionsFeature!.isCaptionsActive) {
        let startCaptionsOptions = StartCaptionsOptions()
        startCaptionsOptions.language = "en-us"
        captionsFeature!.startCaptions(startCaptionsOptions: startCaptionsOptions, completionHandler: { (error) in
            if (error != nil) {
                print ("Call captions Failed to start caption %@", error! as Error)
            }
        })
    }
}

Stopping captions

Remove the previously registered delegate.

func stopCaptions() {
    captionsFeature?.delegate = nil
}

Get available languages

Access the availableLanguages property on the CaptionsCallFeature API.

var captionsAvailableLanguages = captionsFeature!.availableLanguages

Update language

Pass a value in from the available languages array to ensure that the requested language is supported.

func switchCaptionsLanguage() {
    if (!captionsFeature!.isCaptionsActive) {
        return
    }
    captionsFeature!.select(language: captionsAvailableLanguages[0], completionHandler: { (error) in
        if (error != nil) {
            print ("Call captions Failed to switch language %@", error! as Error)
        }
    })
}

Clean up

Learn more about cleaning up resources here.

Clean up resources

If you want to clean up and remove a Communication Services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it. Learn more about cleaning up resources.

Next steps

For more information, see the following articles:

• Learn more about using closed captions in Teams interop scenarios.
• Check out our web calling sample
• Learn about Calling SDK capabilities
• Learn more about how calling works