Manage video during calls

Learn how to manage video calls with the Azure Communication Services SDKS. We'll learn how to manage receiving and sending video within a call.

Prerequisites

Install the SDK

Use the npm install command to install the Azure Communication Services calling and common SDKs for JavaScript.

npm install @azure/communication-common --save
npm install @azure/communication-calling --save

Initialize required objects

A CallClient, instance is required for most call operations. Let's create a new CallClient instance. You can configure it with custom options like a Logger instance.

When you have a CallClient instance, you can create a CallAgent instance by calling the createCallAgent method on the CallClient instance. This method asynchronously returns a CallAgent instance object.

The createCallAgent method uses CommunicationTokenCredential as an argument. It accepts a user access token.

You can use the getDeviceManager method on the CallClient instance to access deviceManager.

const { CallClient } = require('@azure/communication-calling');
const { AzureCommunicationTokenCredential} = require('@azure/communication-common');
const { AzureLogger, setLogLevel } = require("@azure/logger");

// Set the logger's log level
setLogLevel('verbose');

// Redirect log output to wherever desired. To console, file, buffer, REST API, etc...
AzureLogger.log = (...args) => {
    console.log(...args); // Redirect log output to console
};

const userToken = '<USER_TOKEN>';
callClient = new CallClient(options);
const tokenCredential = new AzureCommunicationTokenCredential(userToken);
const callAgent = await callClient.createCallAgent(tokenCredential, {displayName: 'optional Azure Communication Services user name'});
const deviceManager = await callClient.getDeviceManager()

Device management

To begin using video with Calling, you will need to know how to manage devices. Devices allow you to control what transmits Audio and Video to the call.

In deviceManager, you can enumerate local devices that can transmit your audio and video streams in a call. You can also use it to request permission to access the local device's microphones and cameras.

You can access deviceManager by calling the callClient.getDeviceManager() method:

const deviceManager = await callClient.getDeviceManager();

Get local devices

To access local devices, you can use enumeration methods on deviceManager. Enumeration is an asynchronous action

//  Get a list of available video devices for use.
const localCameras = await deviceManager.getCameras(); // [VideoDeviceInfo, VideoDeviceInfo...]

// Get a list of available microphone devices for use.
const localMicrophones = await deviceManager.getMicrophones(); // [AudioDeviceInfo, AudioDeviceInfo...]

// Get a list of available speaker devices for use.
const localSpeakers = await deviceManager.getSpeakers(); // [AudioDeviceInfo, AudioDeviceInfo...]

Set the default microphone and speaker

In deviceManager, you can set a default device that you'll use to start a call. If client defaults aren't set, Communication Services uses operating system defaults.

// Get the microphone device that is being used.
const defaultMicrophone = deviceManager.selectedMicrophone;

// Set the microphone device to use.
await deviceManager.selectMicrophone(localMicrophones[0]);

// Get the speaker device that is being used.
const defaultSpeaker = deviceManager.selectedSpeaker;

// Set the speaker device to use.
await deviceManager.selectSpeaker(localSpeakers[0]);

Local camera preview

You can use deviceManager and VideoStreamRenderer to begin rendering streams from your local camera. This stream won't be sent to other participants; it's a local preview feed.

const cameras = await deviceManager.getCameras();
const camera = cameras[0];
const localVideoStream = new LocalVideoStream(camera);
const videoStreamRenderer = new VideoStreamRenderer(localVideoStream);
const view = await videoStreamRenderer.createView();
htmlElement.appendChild(view.target);

Request permission to camera and microphone

Prompt a user to grant camera and/or microphone permissions:

const result = await deviceManager.askDevicePermission({audio: true, video: true});

This resolves with an object that indicates whether audio and video permissions were granted:

console.log(result.audio);
console.log(result.video);

Notes

  • The 'videoDevicesUpdated' event fires when video devices are plugging-in/unplugged.
  • The 'audioDevicesUpdated' event fires when audio devices are plugged
  • When the DeviceManager is created, at first it does not know about any devices if permissions have not been granted yet, and so initially its device list is empty. If we then call the DeviceManager.askPermission() API, the user is prompted for device access and if the user clicks on 'allow' to grant the access, then the device manager will learn about the devices on the system, update it's device lists and emit the 'audioDevicesUpdated' and 'videoDevicesUpdated' events. Lets say we then refresh the page and create device manager, the device manager will be able to learn about devices because user has already previously granted access, and so it will initially it will have it's device lists filled and it will not emit 'audioDevicesUpdated' nor 'videoDevicesUpdated' events.
  • Speaker enumeration/selection is not supported on Android Chrome, iOS Safari, nor macOS Safari.

Place a call with video camera

Important

Currently only one outgoing local video stream is supported.

To place a video call, you have to enumerate local cameras by using the getCameras() method in deviceManager.

After you select a camera, use it to construct a LocalVideoStream instance. Pass it within videoOptions as an item within the localVideoStream array to the startCall method.

const deviceManager = await callClient.getDeviceManager();
const cameras = await deviceManager.getCameras();
const camera = cameras[0]
const localVideoStream = new LocalVideoStream(camera);
const placeCallOptions = {videoOptions: {localVideoStreams:[localVideoStream]}};
const userCallee = { communicationUserId: '<ACS_USER_ID>' }
const call = callAgent.startCall([userCallee], placeCallOptions);
  • You can also join a call with video with CallAgent.join() API, and accept and call with video with Call.Accept() API.
  • When your call connects, it automatically starts sending a video stream from the selected camera to the other participant.

Start and stop sending local video while on a call

To start a video while on a call, you have to enumerate cameras using the getCameras method on the deviceManager object. Then create a new instance of LocalVideoStream with the desired camera and then pass the LocalVideoStream object into the startVideo method of an existing call object:

const deviceManager = await callClient.getDeviceManager();
const cameras = await deviceManager.getCameras();
const camera = cameras[0]
const localVideoStream = new LocalVideoStream(camera);
await call.startVideo(localVideoStream);

After you successfully start sending video, a LocalVideoStream instance is added to the localVideoStreams collection on a call instance.

call.localVideoStreams[0] === localVideoStream;

To stop local video while on a call, pass the localVideoStream instance that's available in the localVideoStreams collection:

await call.stopVideo(localVideoStream);
// or
await call.stopVideo(call.localVideoStreams[0]);

You can switch to a different camera device while a video is sending by invoking switchSource on a localVideoStream instance:

const cameras = await callClient.getDeviceManager().getCameras();
const camera = cameras[1];
localVideoStream.switchSource(camera);

If the specified video device is being used by another process, or if it is disabled in the system:

  • While in a call, if your video is off and you start video using call.startVideo(), this method will throw with a SourceUnavailableError and cameraStartFiled will be set to true.
  • A call to the localVideoStream.switchSource() method will cause cameraStartFailed to be set to true. Our Call Diagnostics guide provides additional information on how to diagnose call related issues.

To check or verify if the local video is on or off, you can use isLocalVideoStarted API, which returns true or false:

// Check if local video is on or off
call.isLocalVideoStarted;

To listen for changes to the local video, you can subscribe and unsubscribe to the isLocalVideoStartedChanged event

// Subscribe to local video event
call.on('isLocalVideoStartedChanged', () => {
    // Callback();
});
// Unsubscribe from local video event
call.off('isLocalVideoStartedChanged', () => {
    // Callback();
});

Start and stop screen sharing while on a call

To start and stop screen sharing while on a call, you can use asynchronous APIs startScreenSharing and stopScreenSharing respectively:

// Start screen sharing
await call.startScreenSharing();

// Stop screen sharing
await call.stopScreenSharing();

To check or verify if screen sharing is on or off, you can use isScreenSharingOn API which returns true or false:

// Check if screen sharing is on or off
call.isScreenSharingOn;

To listen for changes to the screen share, you can subscribe and unsubscribe to the isScreenSharingOnChanged event

// Subscribe to screen share event
call.on('isScreenSharingOnChanged', () => {
    // Callback();
});
// Unsubscribe from screen share event
call.off('isScreenSharingOnChanged', () => {
    // Callback();
});

Render remote participant video streams

To list the video streams and screen sharing streams of remote participants, inspect the videoStreams collections:

const remoteVideoStream: RemoteVideoStream = call.remoteParticipants[0].videoStreams[0];
const streamType: MediaStreamType = remoteVideoStream.mediaStreamType;

To render RemoteVideoStream, you have to subscribe to it's isAvailableChanged event. If the isAvailable property changes to true, a remote participant is sending a stream. After that happens, create a new instance of VideoStreamRenderer, and then create a new VideoStreamRendererView instance by using the asynchronous createView method. You can then attach view.target to any UI element.

Whenever availability of a remote stream changes, you can choose to destroy the whole VideoStreamRenderer, a specific VideoStreamRendererView or keep them, but this will result in displaying blank video frame.

// Reference to the html's div where we would display a grid of all remote video stream from all participants.
let remoteVideosGallery = document.getElementById('remoteVideosGallery');

subscribeToRemoteVideoStream = async (remoteVideoStream) => {
   let renderer = new VideoStreamRenderer(remoteVideoStream);
    let view;
    let remoteVideoContainer = document.createElement('div');
    remoteVideoContainer.className = 'remote-video-container';

    /**
     * isReceiving API is currently an @beta feature.
     * To use this api please use 'beta' version of Azure Communication Services Calling Web SDK.
     */
    let loadingSpinner = document.createElement('div');
    // See the css example below for styling the loading spinner.
    loadingSpinner.className = 'loading-spinner';
    remoteVideoStream.on('isReceivingChanged', () => {
        try {
            if (remoteVideoStream.isAvailable) {
                const isReceiving = remoteVideoStream.isReceiving;
                const isLoadingSpinnerActive = remoteVideoContainer.contains(loadingSpinner);
                if (!isReceiving && !isLoadingSpinnerActive) {
                    remoteVideoContainer.appendChild(loadingSpinner);
                } else if (isReceiving && isLoadingSpinnerActive) {
                    remoteVideoContainer.removeChild(loadingSpinner);
                }
            }
        } catch (e) {
            console.error(e);
        }
    });

    const createView = async () => {
        // Create a renderer view for the remote video stream.
        view = await renderer.createView();
        // Attach the renderer view to the UI.
        remoteVideoContainer.appendChild(view.target);
        remoteVideosGallery.appendChild(remoteVideoContainer);
    }

    // Remote participant has switched video on/off
    remoteVideoStream.on('isAvailableChanged', async () => {
        try {
            if (remoteVideoStream.isAvailable) {
                await createView();
            } else {
                view.dispose();
                remoteVideosGallery.removeChild(remoteVideoContainer);
            }
        } catch (e) {
            console.error(e);
        }
    });

    // Remote participant has video on initially.
    if (remoteVideoStream.isAvailable) {
        try {
            await createView();
        } catch (e) {
            console.error(e);
        }
    }
    
    console.log(`Initial stream size: height: ${remoteVideoStream.size.height}, width: ${remoteVideoStream.size.width}`);
    remoteVideoStream.on('sizeChanged', () => {
        console.log(`Remote video stream size changed: new height: ${remoteVideoStream.size.height}, new width: ${remoteVideoStream.size.width}`);
    });
}

CSS for styling the loading spinner over the remote video stream.

.remote-video-container {
   position: relative;
}
.loading-spinner {
   border: 12px solid #f3f3f3;
   border-radius: 50%;
   border-top: 12px solid #ca5010;
   width: 100px;
   height: 100px;
   -webkit-animation: spin 2s linear infinite; /* Safari */
   animation: spin 2s linear infinite;
   position: absolute;
   margin: auto;
   top: 0;
   bottom: 0;
   left: 0;
   right: 0;
   transform: translate(-50%, -50%);
}
@keyframes spin {
   0% { transform: rotate(0deg); }
   100% { transform: rotate(360deg); }
}
/* Safari */
@-webkit-keyframes spin {
   0% { -webkit-transform: rotate(0deg); }
   100% { -webkit-transform: rotate(360deg); }
}

Remote video stream properties

Remote video streams have the following properties:

  • id: The ID of a remote video stream.
const id: number = remoteVideoStream.id;
  • mediaStreamType: Can be Video or ScreenSharing.
const type: MediaStreamType = remoteVideoStream.mediaStreamType;
  • isAvailable: Whether a remote participant endpoint is actively sending a stream.
const isAvailable: boolean = remoteVideoStream.isAvailable;
  • isReceiving:

    Note

    This API is provided as a preview for developers and may change based on feedback that we receive. To use this api please use 1.5.4-beta.1+ release of Azure Communication Services Calling Web SDK

    • Will inform the application if remote video stream data is being received or not. Such scenarios are:
      • I am viewing the video of a remote participant who is on mobile browser. The remote participant brings the mobile browser app to the background. I now see the RemoteVideoStream.isReceiving flag goes to false and I see his video with black frames / frozen. When the remote participant brings the mobile browser back to the foreground, I now see the RemoteVideoStream.isReceiving flag to back to true, and I see his video playing normally.
      • I am viewing the video of a remote participant who is on whatever platforms. There are network issues from either side, his video start to look pretty laggy, bad quality, probbaly because of network issues, so I see the RemoteVideoStream.isReceiving flag goes to false.
      • I am viewing the video of a Remote participant who is On MacOS/iOS Safari, and from their address bar, they click on "Pause" / "Resume" camera. I'll see a black/frozen video since they paused their camera and I'll see the RemoteVideoStream.isReceiving flag goes to false. Once they resume playing the camera, then I'll see the RemoteVideoStream.isReceiving flag goes to true.
      • I am viewing the video of a remote participant who in on whatever platform. And for whatever reason their network disconnects. This will actually leave the remote participant in the call for a little while and I'll see his video frozen/black frame, and ill see RemoteVideoStream.isReceiving flag goes to false. The remote participant can get network back and reconnect and his audio/video should start flowing normally and I'll see the RemoteVideoStream.isReceiving flag to true.
      • I am viewing the video of a remote participant who is on mobile browser. The remote participant terminates/kills the mobile browser. Since that remote participant was on mobile, this will actually leave the participant in the call for a little while and I will still see him in the call and his video will be frozen, and so I'll see the RemoteVideoStream.isReceiving flag goes to false. At some point, service will kick participant out of the call and I would just see that the participant disconnected from the call.
      • I am viewing the video of a remote participant who is on mobile browser and they locks device. I'll see the RemoteVideoStream.isReceiving flag goes to false and. Once the remote participant unlocks the device and navigates to the acs call, then ill see the flag go back to true. Same behavior when remote participant is on desktop and the desktop locks/sleeps
    • This feature improves the user experience for rendering remote video streams.
    • You can display a loading spinner over the remote video stream when isReceiving flag changes to false. You don't have to do a loading spinner, you can do anything you desire, but a loading spinner is the most common usage for better user experience.
const isReceiving: boolean = remoteVideoStream.isReceiving;
  • size: The stream size. The higher the stream size, the better the video quality.
const size: StreamSize = remoteVideoStream.size;

VideoStreamRenderer methods and properties

Create a VideoStreamRendererView instance that can be attached in the application UI to render the remote video stream, use asynchronous createView() method, it resolves when stream is ready to render and returns an object with target property that represents video element that can be appended anywhere in the DOM tree

await videoStreamRenderer.createView();

Dispose of videoStreamRenderer and all associated VideoStreamRendererView instances:

videoStreamRenderer.dispose();

VideoStreamRendererView methods and properties

When you create a VideoStreamRendererView, you can specify the scalingMode and isMirrored properties. scalingMode can be Stretch, Crop, or Fit. If isMirrored is specified, the rendered stream is flipped vertically.

const videoStreamRendererView: VideoStreamRendererView = await videoStreamRenderer.createView({ scalingMode, isMirrored });

Every VideoStreamRendererView instance has a target property that represents the rendering surface. Attach this property in the application UI:

htmlElement.appendChild(view.target);

You can update scalingMode by invoking the updateScalingMode method:

view.updateScalingMode('Crop');

Install the SDK

Locate your project level build.gradle and make sure to add mavenCentral() to the list of repositories under buildscript and allprojects

buildscript {
    repositories {
    ...
        mavenCentral()
    ...
    }
}
allprojects {
    repositories {
    ...
        mavenCentral()
    ...
    }
}

Then, in your module level build.gradle add the following lines to the dependencies section

dependencies {
    ...
    implementation 'com.azure.android:azure-communication-calling:1.0.0'
    ...
}

Initialize the required objects

To create a CallAgent instance you have to call the createCallAgent method on a CallClient instance. This asynchronously returns a CallAgent instance object. The createCallAgent method takes a CommunicationUserCredential as an argument, which encapsulates an access token. To access the DeviceManager, a callAgent instance must be created first, and then you can use the CallClient.getDeviceManager method to get the DeviceManager.

String userToken = '<user token>';
CallClient callClient = new CallClient();
CommunicationTokenCredential tokenCredential = new CommunicationTokenCredential(userToken);
android.content.Context appContext = this.getApplicationContext(); // From within an Activity for instance
CallAgent callAgent = callClient.createCallAgent(appContext, tokenCredential).get();
DeviceManager deviceManager = callClient.getDeviceManager(appContext).get();

To set a display name for the caller, use this alternative method:

String userToken = '<user token>';
CallClient callClient = new CallClient();
CommunicationTokenCredential tokenCredential = new CommunicationTokenCredential(userToken);
android.content.Context appContext = this.getApplicationContext(); // From within an Activity for instance
CallAgentOptions callAgentOptions = new CallAgentOptions();
callAgentOptions.setDisplayName("Alice Bob");
DeviceManager deviceManager = callClient.getDeviceManager(appContext).get();
CallAgent callAgent = callClient.createCallAgent(appContext, tokenCredential, callAgentOptions).get();

Device management

To begin using video with Calling, you will need to know how to manage devices. Devices allow you to control what transmits Audio and Video to the call.

DeviceManager lets you enumerate local devices that can be used in a call to transmit your audio/video streams. It also allows you to request permission from a user to access their microphone and camera using the native browser API.

You can access deviceManager by calling callClient.getDeviceManager() method.

Context appContext = this.getApplicationContext();
DeviceManager deviceManager = callClient.getDeviceManager(appContext).get();

Enumerate local devices

To access local devices, you can use enumeration methods on the Device Manager. Enumeration is a synchronous action.

//  Get a list of available video devices for use.
List<VideoDeviceInfo> localCameras = deviceManager.getCameras(); // [VideoDeviceInfo, VideoDeviceInfo...]

Local camera preview

You can use DeviceManager and Renderer to begin rendering streams from your local camera. This stream won't be sent to other participants; it's a local preview feed. This is an asynchronous action.

VideoDeviceInfo videoDevice = <get-video-device>; // See the `Enumerate local devices` topic above
Context appContext = this.getApplicationContext();

LocalVideoStream currentVideoStream = new LocalVideoStream(videoDevice, appContext);

LocalVideoStream[] localVideoStreams = new LocalVideoStream[1];
localVideoStreams[0] = currentVideoStream;

VideoOptions videoOptions = new VideoOptions(localVideoStreams);

RenderingOptions renderingOptions = new RenderingOptions(ScalingMode.Fit);
VideoStreamRenderer previewRenderer = new VideoStreamRenderer(currentVideoStream, appContext);

VideoStreamRendererView uiView = previewRenderer.createView(renderingOptions);

// Attach the uiView to a viewable location on the app at this point
layout.addView(uiView);

Place a 1:1 call with video camera

Warning

Currently only one outgoing local video stream is supported To place a call with video you have to enumerate local cameras using the deviceManager getCameras API. Once you select a desired camera, use it to construct a LocalVideoStream instance and pass it into videoOptions as an item in the localVideoStream array to a call method. Once the call connects it will automatically start sending a video stream from the selected camera to other participant(s).

Note

Due to privacy concerns, video will not be shared to the call if it is not being previewed locally. See Local camera preview for more details.

VideoDeviceInfo desiredCamera = <get-video-device>; // See the `Enumerate local devices` topic above
Context appContext = this.getApplicationContext();

LocalVideoStream currentVideoStream = new LocalVideoStream(desiredCamera, appContext);

LocalVideoStream[] localVideoStreams = new LocalVideoStream[1];
localVideoStreams[0] = currentVideoStream;

VideoOptions videoOptions = new VideoOptions(localVideoStreams);

// Render a local preview of video so the user knows that their video is being shared
Renderer previewRenderer = new VideoStreamRenderer(currentVideoStream, appContext);
View uiView = previewRenderer.createView(new CreateViewOptions(ScalingMode.FIT));

// Attach the uiView to a viewable location on the app at this point
layout.addView(uiView);

CommunicationUserIdentifier[] participants = new CommunicationUserIdentifier[]{ new CommunicationUserIdentifier("<acs user id>") };

StartCallOptions startCallOptions = new StartCallOptions();
startCallOptions.setVideoOptions(videoOptions);

Call call = callAgent.startCall(context, participants, startCallOptions);

Start and stop sending local video

To start a video, you have to enumerate cameras using the getCameraList API on deviceManager object. Then create a new instance of LocalVideoStream passing the desired camera, and pass it in the startVideo API as an argument:

VideoDeviceInfo desiredCamera = <get-video-device>; // See the `Enumerate local devices` topic above
Context appContext = this.getApplicationContext();

LocalVideoStream currentLocalVideoStream = new LocalVideoStream(desiredCamera, appContext);

VideoOptions videoOptions = new VideoOptions(currentLocalVideoStream);

Future startVideoFuture = call.startVideo(appContext, currentLocalVideoStream);
startVideoFuture.get();

Once you successfully start sending video, a LocalVideoStream instance will be added to the localVideoStreams collection on the call instance.

List<LocalVideoStream> videoStreams = call.getLocalVideoStreams();
LocalVideoStream currentLocalVideoStream = videoStreams.get(0); // Please make sure there are VideoStreams in the list before calling get(0).

To stop local video, pass the LocalVideoStream instance available in localVideoStreams collection:

call.stopVideo(appContext, currentLocalVideoStream).get();

You can switch to a different camera device while video is being sent by invoking switchSource on a LocalVideoStream instance:

currentLocalVideoStream.switchSource(source).get();

Render remote participant video streams

To list the video streams and screen sharing streams of remote participants, inspect the videoStreams collections:

List<RemoteParticipant> remoteParticipants = call.getRemoteParticipants();
RemoteParticipant remoteParticipant = remoteParticipants.get(0); // Please make sure there are remote participants in the list before calling get(0).

List<RemoteVideoStream> remoteStreams = remoteParticipant.getVideoStreams();
RemoteVideoStream remoteParticipantStream = remoteStreams.get(0); // Please make sure there are video streams in the list before calling get(0).

MediaStreamType streamType = remoteParticipantStream.getType(); // of type MediaStreamType.Video or MediaStreamType.ScreenSharing

To render a RemoteVideoStream from a remote participant, you have to subscribe to a OnVideoStreamsUpdated event.

Within the event, the change of isAvailable property to true indicates that remote participant is currently sending a stream. Once that happens, create new instance of a Renderer, then create a new RendererView using asynchronous createView API and attach view.target anywhere in the UI of your application.

Whenever availability of a remote stream changes you can choose to destroy the whole Renderer, a specific RendererView or keep them, but this will result in displaying blank video frame.

VideoStreamRenderer remoteVideoRenderer = new VideoStreamRenderer(remoteParticipantStream, appContext);
VideoStreamRendererView uiView = remoteVideoRenderer.createView(new RenderingOptions(ScalingMode.FIT));
layout.addView(uiView);

remoteParticipant.addOnVideoStreamsUpdatedListener(e -> onRemoteParticipantVideoStreamsUpdated(p, e));

void onRemoteParticipantVideoStreamsUpdated(RemoteParticipant participant, RemoteVideoStreamsEvent args) {
    for(RemoteVideoStream stream : args.getAddedRemoteVideoStreams()) {
        if(stream.getIsAvailable()) {
            startRenderingVideo();
        } else {
            renderer.dispose();
        }
    }
}

Remote video stream properties

Remote video stream has couple of properties

  • Id - ID of a remote video stream
int id = remoteVideoStream.getId();
  • MediaStreamType - Can be 'Video' or 'ScreenSharing'
MediaStreamType type = remoteVideoStream.getMediaStreamType();
  • isAvailable - Indicates if remote participant endpoint is actively sending stream
boolean availability = remoteVideoStream.isAvailable();

Renderer methods and properties

Renderer object following APIs

  • Create a VideoStreamRendererView instance that can be later attached in the application UI to render remote video stream.
// Create a view for a video stream
VideoStreamRendererView.createView()
  • Dispose renderer and all VideoStreamRendererView associated with this renderer. To be called when you have removed all associated views from the UI.
VideoStreamRenderer.dispose()
  • StreamSize - size (width/height) of a remote video stream
StreamSize renderStreamSize = VideoStreamRenderer.getSize();
int width = renderStreamSize.getWidth();
int height = renderStreamSize.getHeight();

RendererView methods and properties

When creating a VideoStreamRendererView you can specify the ScalingMode and mirrored properties that will apply to this view: Scaling mode can be either of 'CROP' | 'FIT'

VideoStreamRenderer remoteVideoRenderer = new VideoStreamRenderer(remoteVideoStream, appContext);
VideoStreamRendererView rendererView = remoteVideoRenderer.createView(new CreateViewOptions(ScalingMode.Fit));

The created RendererView can then be attached to the application UI using the following snippet:

layout.addView(rendererView);

You can later update the scaling mode by invoking updateScalingMode API on the RendererView object with one of ScalingMode.CROP | ScalingMode.FIT as an argument.

// Update the scale mode for this view.
rendererView.updateScalingMode(ScalingMode.CROP)

Set up your system

Create the Xcode project

In Xcode, create a new iOS project and select the Single View App template. This quickstart uses the SwiftUI framework, so you should set the Language to Swift and User Interface to SwiftUI.

You're not going to create unit tests or UI tests during this quickstart. Feel free to clear the Include Unit Tests and Include UI Tests text boxes.

Screenshot that shows the window for creating a project within Xcode.

Install the package and dependencies with CocoaPods

  1. Create a Podfile for your application, like this:

    platform :ios, '13.0'
    use_frameworks!
    target 'AzureCommunicationCallingSample' do
        pod 'AzureCommunicationCalling', '~> 1.0.0'
    end
    
  2. Run pod install.

  3. Open .xcworkspace with Xcode.

Request access to the microphone

To access the device's microphone, you need to update your app's information property list with NSMicrophoneUsageDescription. You set the associated value to a string that will be included in the dialog that the system uses to request access from the user.

Right-click the Info.plist entry of the project tree and select Open As > Source Code. Add the following lines in the top-level <dict> section, and then save the file.

<key>NSMicrophoneUsageDescription</key>
<string>Need microphone access for VOIP calling.</string>

Set up the app framework

Open your project's ContentView.swift file and add an import declaration to the top of the file to import the AzureCommunicationCalling library. In addition, import AVFoundation. You'll need it for audio permission requests in the code.

import AzureCommunicationCalling
import AVFoundation

Initialize CallAgent

To create a CallAgent instance from CallClient, you have to use a callClient.createCallAgent method that asynchronously returns a CallAgent object after it's initialized.

To create a call client, you have to pass a CommunicationTokenCredential object.

import AzureCommunication

let tokenString = "token_string"
var userCredential: CommunicationTokenCredential?
do {
    let options = CommunicationTokenRefreshOptions(initialToken: token, refreshProactively: true, tokenRefresher: self.fetchTokenSync)
    userCredential = try CommunicationTokenCredential(withOptions: options)
} catch {
    updates("Couldn't created Credential object", false)
    initializationDispatchGroup!.leave()
    return
}

// tokenProvider needs to be implemented by Contoso, which fetches a new token
public func fetchTokenSync(then onCompletion: TokenRefreshOnCompletion) {
    let newToken = self.tokenProvider!.fetchNewToken()
    onCompletion(newToken, nil)
}

Pass the CommunicationTokenCredential object that you created to CallClient, and set the display name.

self.callClient = CallClient()
let callAgentOptions = CallAgentOptions()
options.displayName = " iOS Azure Communication Services User"

self.callClient!.createCallAgent(userCredential: userCredential!,
    options: callAgentOptions) { (callAgent, error) in
        if error == nil {
            print("Create agent succeeded")
            self.callAgent = callAgent
        } else {
            print("Create agent failed")
        }
})

Manage devices

To begin using video with Calling, you will need to know how to manage devices. Devices allow you to control what transmits Audio and Video to the call.

DeviceManager lets you enumerate local devices that can be used in a call to transmit audio or video streams. It also allows you to request permission from a user to access a microphone or camera. You can access deviceManager on the callClient object.

self.callClient!.getDeviceManager { (deviceManager, error) in
        if (error == nil) {
            print("Got device manager instance")
            self.deviceManager = deviceManager
        } else {
            print("Failed to get device manager instance")
        }
    }

Enumerate local devices

To access local devices, you can use enumeration methods on the device manager. Enumeration is a synchronous action.

// enumerate local cameras
var localCameras = deviceManager.cameras // [VideoDeviceInfo, VideoDeviceInfo...]

Get a local camera preview

You can use Renderer to begin rendering a stream from your local camera. This stream won't be sent to other participants; it's a local preview feed. This is an asynchronous action.

let camera: VideoDeviceInfo = self.deviceManager!.cameras.first!
let localVideoStream = LocalVideoStream(camera: camera)
let localRenderer = try! VideoStreamRenderer(localVideoStream: localVideoStream)
self.view = try! localRenderer.createView()

Get local camera preview properties

The renderer has set of properties and methods that allow you to control the rendering.

// Constructor can take in LocalVideoStream or RemoteVideoStream
let localRenderer = VideoStreamRenderer(localVideoStream:localVideoStream)
let remoteRenderer = VideoStreamRenderer(remoteVideoStream:remoteVideoStream)

// [StreamSize] size of the rendering view
localRenderer.size

// [VideoStreamRendererDelegate] an object you provide to receive events from this Renderer instance
localRenderer.delegate

// [Synchronous] create view
try! localRenderer.createView()

// [Synchronous] create view with rendering options
try! localRenderer!.createView(withOptions: CreateViewOptions(scalingMode: ScalingMode.fit))

// [Synchronous] dispose rendering view
localRenderer.dispose()

Place a 1:1 call with video

To get a device manager instance, see the section about managing devices.

let firstCamera = self.deviceManager!.cameras.first
self.localVideoStreams = [LocalVideoStream]()
self.localVideoStreams!.append(LocalVideoStream(camera: firstCamera!))
let videoOptions = VideoOptions(localVideoStreams: self.localVideoStreams!)

let startCallOptions = StartCallOptions()
startCallOptions.videoOptions = videoOptions

let callee = CommunicationUserIdentifier('UserId')
self.callAgent?.startCall(participants: [callee], options: startCallOptions) { (call, error) in
    if error == nil {
        print("Successfully started outgoing video call")
        self.call = call
    } else {
        print("Failed to start outgoing video call")
    }
}

Render remote participant video streams

Remote participants can initiate video or screen sharing during a call.

Handle video-sharing or screen-sharing streams of remote participants

To list the streams of remote participants, inspect the videoStreams collections.

var remoteParticipantVideoStream = call.remoteParticipants[0].videoStreams[0]

Get remote video stream properties

var type: MediaStreamType = remoteParticipantVideoStream.type // 'MediaStreamTypeVideo'
var isAvailable: Bool = remoteParticipantVideoStream.isAvailable // indicates if remote stream is available
var id: Int = remoteParticipantVideoStream.id // id of remoteParticipantStream

Render remote participant streams

To start rendering remote participant streams, use the following code.

let renderer = VideoStreamRenderer(remoteVideoStream: remoteParticipantVideoStream)
let targetRemoteParticipantView = renderer?.createView(withOptions: CreateViewOptions(scalingMode: ScalingMode.crop))
// To update the scaling mode later
targetRemoteParticipantView.update(scalingMode: ScalingMode.fit)

Get remote video renderer methods and properties

// [Synchronous] dispose() - dispose renderer and all `RendererView` associated with this renderer. To be called when you have removed all associated views from the UI.
remoteVideoRenderer.dispose()

Setting up

Creating the Visual Studio project

For UWP app, in Visual Studio 2019, create a new Blank App (Universal Windows) project. After entering the project name, feel free to pick any Windows SDK greater than 10.0.17134.

For WinUI 3 app, create a new project with the Blank App, Packaged (WinUI 3 in Desktop) template to set up a single-page WinUI 3 app. Windows App SDK version 1.2 preview 2 and above is required.

Install the package and dependencies with NuGet Package Manager

The Calling SDK APIs and libraries are publicly available via a NuGet package. The following steps exemplify how to find, download, and install the Calling SDK NuGet package.

  1. Open NuGet Package Manager (Tools -> NuGet Package Manager -> Manage NuGet Packages for Solution)
  2. Click on Browse and then type Azure.Communication.Calling in the search box.
  3. Make sure that Include prerelease check box is selected.
  4. Click on the Azure.Communication.Calling package, select Azure.Communication.Calling 1.0.0-beta.33 or newer version.
  5. Select the checkbox corresponding to the CS project on the right-side tab.
  6. Click on the Install button.

Request access to the microphone

The app will require access to the camera to run properly. In UWP apps, the camera capability should be declared in the app manifest file. he following steps exemplify how to achieve that.

  1. In the Solution Explorer panel, double click on the file with .appxmanifest extension.
  2. Click on the Capabilities tab.
  3. Select the Camera check box from the capabilities list.

Create UI buttons to place and hang up the call

This simple sample app will contain two buttons. One for placing the call and another to hang up a placed call. The following steps exemplify how to add these buttons to the app.

  1. In the Solution Explorer panel, double click on the file named MainPage.xaml for UWP, or MainWindows.xaml for WinUI 3.
  2. In the central panel, look for the XAML code under the UI preview.
  3. Modify the XAML code by the following excerpt:
<TextBox x:Name="CalleeTextBox" Text="Who would you like to call?" TextWrapping="Wrap" VerticalAlignment="Center" Grid.Row="0" Height="40" Margin="10,10,10,10" />
<StackPanel Orientation="Horizontal">
    <Button x:Name="CallButton" Content="Start Call" Click="CallButton_Click" VerticalAlignment="Center" Margin="10,0,0,0" Height="40" Width="200"/>
    <Button x:Name="HangupButton" Content="Hang Up" Click="HangupButton_Click" VerticalAlignment="Center" Margin="10,0,0,0" Height="40" Width="200"/>
</StackPanel>

Setting up the app with Calling SDK APIs

The Calling SDK APIs are in two different namespaces. The following steps inform the C# compiler about these namespaces allowing Visual Studio's Intellisense to assist with code development.

  1. In the Solution Explorer panel, click on the arrow on the left side of the file named MainPage.xaml for UWP, or MainWindows.xaml for WinUI 3.
  2. Double click on file named MainPage.xaml.cs or MainWindows.xaml.cs.
  3. Add the following commands at the bottom of the current using statements.
using Azure.Communication;
using Azure.Communication.Calling;

Please keep MainPage.xaml.cs or MainWindows.xaml.cs open. The next steps will add more code to it.

Allow app interactions

The UI buttons previously added need to operate on top of a placed Call. It means that a Call data member should be added to the MainPage or MainWindow class. Additionally, to allow the asynchronous operation creating CallAgent to succeed, a CallAgent data member should also be added to the same class.

Please add the following data members to the MainPage or MainWindow class:

CallAgent callAgent;
Call call;

Create button handlers

Previously, two UI buttons were added to the XAML code. The following code adds the handlers to be executed when a user selects the button. The following code should be added after the data members from the previous section.

private async void CallButton_Click(object sender, RoutedEventArgs e)
{
    // Start call
}

private async void HangupButton_Click(object sender, RoutedEventArgs e)
{
    // End the current call
}

Object model

The following classes and interfaces handle some of the major features of the Azure Communication Services Calling client library for UWP.

Name Description
CallClient The CallClient is the main entry point to the Calling client library.
CallAgent The CallAgent is used to start and join calls.
Call The Call is used to manage placed or joined calls.
CommunicationTokenCredential The CommunicationTokenCredential is used as the token credential to instantiate the CallAgent.
CallAgentOptions The CallAgentOptions contains information to identify the caller.
HangupOptions The HangupOptions informs if a call should be terminated to all its participants.

Register video handler

A UI component, like XAML's MediaElement or MediaPlayerElement, will require the app registering a configuration for rendering local and remote video feeds. Please add the following content between the Package tags of the Package.appxmanifest:

<Extensions>
    <Extension Category="windows.activatableClass.inProcessServer">
        <InProcessServer>
            <Path>RtmMvrUap.dll</Path>
            <ActivatableClass ActivatableClassId="VideoN.VideoSchemeHandler" ThreadingModel="both" />
        </InProcessServer>
    </Extension>
</Extensions>

Initialize the CallAgent

To create a CallAgent instance from CallClient you must use CallClient.CreateCallAgent method that asynchronously returns a CallAgent object once it is initialized.

To create CallAgent, you must pass a CommunicationTokenCredential object and a CallAgentOptions object. Keep in mind that CommunicationTokenCredential throws if a malformed token is passed.

The following code should be added inside and helper function to be called in app initialization.

var callClient = new CallClient();
this.deviceManager = await callClient.GetDeviceManager();

var tokenCredential = new CommunicationTokenCredential("<AUTHENTICATION_TOKEN>");
var callAgentOptions = new CallAgentOptions()
{
    DisplayName = "<DISPLAY_NAME>"
};

this.callAgent = await callClient.CreateCallAgent(tokenCredential, callAgentOptions);
this.callAgent.OnCallsUpdated += Agent_OnCallsUpdatedAsync;
this.callAgent.OnIncomingCall += Agent_OnIncomingCallAsync;

<AUTHENTICATION_TOKEN> must be replaced by a valid credential token for your resource. Refer to the user access token documentation if a credential token has to be sourced.

Place a 1:1 call with video camera

The objects needed for creating a CallAgent are now ready. It is time to asynchronously create CallAgent and place a video call.

The following code should be added after handling the exception from the previous step.

var startCallOptions = new StartCallOptions();

if ((LocalVideo.Source == null) && (this.deviceManager.Cameras?.Count > 0))
{
    var videoDeviceInfo = this.deviceManager.Cameras?.FirstOrDefault();
    if (videoDeviceInfo != null)
    {
        // <Initialize local camera preview>
        startCallOptions.VideoOptions = new VideoOptions(new[] { localVideoStream });
    }
}

var callees = new ICommunicationIdentifier[1] { new CommunicationUserIdentifier(CalleeTextBox.Text.Trim()) };

this.call = await this.callAgent.StartCallAsync(callees, startCallOptions);
this.call.OnRemoteParticipantsUpdated += Call_OnRemoteParticipantsUpdatedAsync;
this.call.OnStateChanged += Call_OnStateChangedAsync;

Local camera preview

We can optionally set up local camera preview. The video can be rendered through UWP MediaElement:

<Grid Grid.Row="1">
    <Grid.RowDefinitions>
        <RowDefinition/>
    </Grid.RowDefinitions>
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width="*"/>
        <ColumnDefinition Width="*"/>
    </Grid.ColumnDefinitions>
    <MediaElement x:Name="LocalVideo" HorizontalAlignment="Center" Stretch="UniformToFill" Grid.Column="0" VerticalAlignment="Center"/>
    <MediaElement x:Name="RemoteVideo" HorizontalAlignment="Center" Stretch="UniformToFill" Grid.Column="1" VerticalAlignment="Center"/>
</Grid>

To initialize the local preview MedialElement:

var localVideoStream = new LocalVideoStream(videoDeviceInfo);

var localUri = await localVideoStream.MediaUriAsync();

await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
{
    LocalVideo.Source = localUri;
    LocalVideo.Play();
});

Or, by MediaPlayerElement in WinUI 3:

<Grid Grid.Row="1">
    <Grid.RowDefinitions>
        <RowDefinition/>
    </Grid.RowDefinitions>
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width="*"/>
        <ColumnDefinition Width="*"/>
    </Grid.ColumnDefinitions>
    <MediaPlayerElement x:Name="LocalVideo" HorizontalAlignment="Center" Stretch="UniformToFill" Grid.Column="0" VerticalAlignment="Center"/>
    <MediaPlayerElement x:Name="RemoteVideo" HorizontalAlignment="Center" Stretch="UniformToFill" Grid.Column="1" VerticalAlignment="Center"/>
</Grid>

To initialize the local preview MediaPlayerElement:

var videoDeviceInfo = this.deviceManager.Cameras?.FirstOrDefault();
if (videoDeviceInfo != null)
{
    var localVideoStream = new LocalVideoStream(videoDeviceInfo);

    var localUri = await localVideoStream.MediaUriAsync();

    this.DispatcherQueue.TryEnqueue(() => {
        LocalVideo.Source = MediaSource.CreateFromUri(localUri);
        LocalVideo.MediaPlayer.Play();
    });
}

Render remote camera stream

Set up even handler in response to OnCallsUpdated event:

private async void Agent_OnCallsUpdatedAsync(object sender, CallsUpdatedEventArgs args)
{
    foreach (var call in args.AddedCalls)
    {
        foreach (var remoteParticipant in call.RemoteParticipants)
        {
            var remoteParticipantMRI = remoteParticipant.Identifier.ToString();
            this.remoteParticipantDictionary.TryAdd(remoteParticipantMRI, remoteParticipant);
            await AddVideoStreamsAsync(remoteParticipant.VideoStreams);
            remoteParticipant.OnVideoStreamsUpdated += Call_OnVideoStreamsUpdatedAsync;
        }
    }
}

Start rendering remote video stream on MediaElement for UWP app:

private async Task AddVideoStreamsAsync(IReadOnlyList<RemoteVideoStream> remoteVideoStreams)
{
    foreach (var remoteVideoStream in remoteVideoStreams)
    {
        var remoteUri = await remoteVideoStream.Start();

        await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
        {
            RemoteVideo.Source = remoteUri;
            RemoteVideo.Play();
        });
    }
}

Or, render remote video stream on MediaPlayerElement for Win32 3 app:

private async Task AddVideoStreamsAsync(IReadOnlyList<RemoteVideoStream> remoteVideoStreams)
{
    foreach (var remoteVideoStream in remoteVideoStreams)
    {
        var remoteUri = await remoteVideoStream.Start();

        this.DispatcherQueue.TryEnqueue(() => {
            RemoteVideo.Source = MediaSource.CreateFromUri(remoteUri);
            RemoteVideo.MediaPlayer.Play();
        });
    }
}

End a call

Once a call is placed, the HangupAsync method of the Call object should be used to hang up the call.

An instance of HangupOptions should also be used to inform if the call must be terminated to all its participants.

The following code should be added inside HangupButton_Click.

this.call.OnStateChanged -= Call_OnStateChangedAsync;
await this.call.HangUpAsync(new HangUpOptions());

Run the code

Make sure Visual Studio will build the app for x64, x86 or ARM64, then hit F5 to start running the app. After that, click on the Call button to place a call to the callee defined.

Keep in mind that the first time the app runs, the system will prompt user for granting access to the microphone.

Next steps