QuickStart: Add video effects to your video calls

Artikkel
07/19/2023

You can use the Video effects feature to add effects to your video in video calls. This feature enables developers to build background visual effects and background video replacement into the calling experience. Background blur provides users with the mechanism to remove distractions behind a participant so that participants can communicate without disruptive activity or confidential information in the background. This is especially useful the context of telehealth, where a provider or patient might want to obscure their surroundings to protect sensitive information or personally identifiable information. Background blur can be applied across all virtual appointment scenarios, including telebanking and virtual hearings, to protect user privacy. In addition to enhanced confidentiality, background blur allows for more creativity of expression, allowing users to upload custom backgrounds to host a more fun, personalized calling experience.

Note

The calling effect library cannot be used standalone and can only work when used with the Azure Communication Calling client library for WebJS (https://www.npmjs.com/package/@azure/communication-calling).

Using video effects

Install the package

Use the npm install command to install the Azure Communication Services Effects SDK for JavaScript.

Important

This quickstart uses the Azure Communication Services Calling SDK version of 1.13.1 (or greater) and the Azure Communication Services Calling Effects SDK version greater than or equil to 1.0.1.

npm install @azure/communication-calling-effects --save

See here for more details on the calling commmunication effects npm package page.

Note

Currently browser support for creating video background effects is only supported on Chrome and Edge Desktop Browser (Windows and Mac) and Mac Safari Desktop.

Note

Currently there are two available video effects:

Background blur
Background replacement with an image (the aspect ratio should be 16:9 to be compatible)

To use video effects with the Azure Communication Calling SDK, once you've created a LocalVideoStream, you need to get the VideoEffects feature API of the LocalVideoStream to start/stop video effects:

import * as AzureCommunicationCallingSDK from '@azure/communication-calling'; 

import { BackgroundBlurEffect, BackgroundReplacementEffect } from '@azure/communication-calling-effects'; 

// Get the video effects feature api on the LocalVideoStream 
// (here, localVideoStream is the LocalVideoStream object you created while setting up video calling)
const videoEffectsFeatureApi = localVideoStream.feature(AzureCommunicationCallingSDK.Features.VideoEffects); 


// Subscribe to useful events 
videoEffectsFeatureApi.on(‘effectsStarted’, () => { 
    // Effects started
});

videoEffectsFeatureApi.on(‘effectsStopped’, () => { 
    // Effects stopped
}); 

videoEffectsFeatureApi.on(‘effectsError’, (error) => { 
    // Effects error
});

Background blur

// Create the effect instance 
const backgroundBlurEffect = new BackgroundBlurEffect(); 

// Recommended: Check support by using the isSupported method on the feature API
const backgroundBlurSupported = await videoEffectsFeatureApi.isSupported(backgroundBlurEffect);

if (backgroundBlurSupported) { 
    // Use the video effects feature api we created to start effects
    await videoEffectsFeatureApi.startEffects(backgroundBlurEffect); 
}

Background replacement with an image

You need to provide the URL of the image you want as the background to this effect.

Important

The startEffects method will fail if the URL is not of an image or is unreachable/unreadable.

Note

Current supported image formats are: png, jpg, jpeg, tiff, bmp.

Current supported aspect ratio is 16:9.

const backgroundImage = 'https://linkToImageFile'; 

// Create the effect instance 
const backgroundReplacementEffect = new BackgroundReplacementEffect({ 
    backgroundImageUrl: backgroundImage
}); 

// Recommended: Check support by using the isSupported method on the feature API
const backgroundReplacementSupported = await videoEffectsFeatureApi.isSupported(backgroundReplacementEffect);

if (backgroundReplacementSupported) { 
    // Use the video effects feature api as before to start/stop effects 
    await videoEffectsFeatureApi.startEffects(backgroundReplacementEffect); 
}

Changing the image for this effect can be done by passing it via the configure method:

const newBackgroundImage = 'https://linkToNewImageFile'; 

await backgroundReplacementEffect.configure({ 
    backgroundImageUrl: newBackgroundImage
});

Switching effects can be done using the same method on the video effects feature api:

// Switch to background blur 
await videoEffectsFeatureApi.startEffects(backgroundBlurEffect); 

// Switch to background replacement 
await videoEffectsFeatureApi.startEffects(backgroundReplacementEffect);

At anytime if you want to check what effects are active, you can use the activeEffects property. The activeEffects property returns an array with the names of the current active effects, and returns an empty array if there are no effects active.

// Using the video effects feature api
const currentActiveEffects = videoEffectsFeatureApi.activeEffects;

To stop effects:

await videoEffectsFeatureApi.stopEffects();

Note

In order to use Video Effects on the iOS Calling SDK, a machine learning model is downloaded to the customer's device. We encourage you to review the privacy notes in your application and update them accordingly, if necessary.

You can use the Video Effects feature to add effects to your video in video calls. Background blur provides users with the mechanism to remove distractions behind a participant so that participants can communicate without disruptive activity or confidential information in the background. This feature is especially useful the context of telehealth, where a provider or patient might want to obscure their surroundings to protect sensitive information or personal data. Background blur can be applied across all virtual appointment scenarios, including telebanking and virtual hearings, to protect user privacy.

Using video effects

Note

Video effects support on iOS is limited to the two most recent major versions of iOS. For example, when a new, major version of iOS is released, the iOS requirement is the new version and the most recent versions that preceded it.

Currently there's one available Video Effect: Background Blur.

The LocalVideoEffectsFeature object has the following API structure:

enable: Enables a Video Effect on the LocalVideoStream instance.
disable: Disables a Video Effect on the LocalVideoStream instance.
isSupported: Indicates if a Video Effect is supported on the LocalVideoStream instance.
onVideoEffectEnabled: Event that is triggered when a Video Effect has been enabled successfully.
onVideoEffectDisabled: Event that is triggered when a Video Effect has been disabled successfully.
onVideoEffectError: Event that is triggered when a Video Effect operation fails.

Once you have the LocalVideoEffectsFeature object, you can subscribe to the events, events have the following delegates: didEnableVideoEffect, didDisableVideoEffect, didReceiveVideoEffectError.

To use Video Effects with the Azure Communication Calling SDK, once you've created a LocalVideoStream, you need to get the VideoEffects feature API of the LocalVideoStream to enable/disable Video Effects:

// Obtain the Video Effects feature from the LocalVideoStream object that is sending the video.
@State var localVideoEffectsFeature: LocalVideoEffectsFeature?
localVideoEffectsFeature = self.localVideoStreams.first.feature(Features.localVideoEffects)

// Subscribe to the events
public func localVideoEffectsFeature(_ videoEffectsLocalVideoStreamFeature: LocalVideoEffectsFeature, didEnableVideoEffect args: VideoEffectEnabledEventArgs) {
        os_log("Video Effect Enabled, VideoEffectName: %s", log:log, args.videoEffectName)
    }
public func localVideoEffectsFeature(_ videoEffectsLocalVideoStreamFeature: LocalVideoEffectsFeature, didDisableVideoEffect args: VideoEffectDisabledEventArgs) {
    os_log("Video Effect Disabled, VideoEffectName: %s", log:log, args.videoEffectName)
}
public func localVideoEffectsFeature(_ videoEffectsLocalVideoStreamFeature: LocalVideoEffectsFeature, didReceiveVideoEffectError args: VideoEffectErrorEventArgs) {
    os_log("Video Effect Error, VideoEffectName: %s, Code: %s, Message: %s", log:log, args.videoEffectName, args.code, args.message)
}

and start using the APIs to enable and disable Video Effects:

localVideoEffectsFeature.enable(effect: backgroundBlurVideoEffect)
localVideoEffectsFeature.disable(effect: backgroundBlurVideoEffect)

Background blur

Background Blur is a Video Effect that allows a person's background to be blurred. In order to use Background Video Effect, you need to obtain a LocalVideoEffectsFeature feature from a valid LocalVideoStream.

Create a new Background Blur Video Effect object:

@State var backgroundBlurVideoEffect: BackgroundBlurEffect?

Check if BackgroundBlurEffect is supported and call Enable on the videoEffectsFeature object:

if((localVideoEffectsFeature.isSupported(effect: backgroundBlurVideoEffect)) != nil)
{
    localVideoEffectsFeature.enable(effect: backgroundBlurVideoEffect)
}

To disable Background Blur Video Effect:

localVideoEffectsFeature.disable(effect: backgroundBlurVideoEffect)

Background Replacement

Background Replacement is a Video Effect that allows a person to set their own custom background. In order to use Background Replacement Effect, you need to obtain a LocalVideoEffectsFeature feature from a valid LocalVideoStream.

Create a new Background Replacement Video Effect object:

@State var backgroundReplacementVideoEffect: BackgroundReplacementEffect?

Set a custom background by passing in the image through a buffer.

let image = UIImage(named:"image.png")
guard let imageData = image?.jpegData(compressionQuality: 1.0) else {
return
}
backgroundReplacementVideoEffect.buffer = imageData

Check if BackgroundReplacementEffect is supported and call Enable on the videoEffectsFeature object:

if((localVideoEffectsFeature.isSupported(effect: backgroundReplacementVideoEffect)) != nil)
{
    localVideoEffectsFeature.enable(effect: backgroundReplacementVideoEffect)
}

To disable Background Replacement Video Effect:

localVideoEffectsFeature.disable(effect: backgroundReplacementVideoEffect)

Note

In order to use Video Effects on the Android Calling SDK, a machine learning model is downloaded to the customer's device. We encourage you to review the privacy notes in your application and update them accordingly, if necessary.

This quickstart builds on Quickstart: Add 1:1 video calling to your app for Android.

Using video effects

Note

Video effects support on Android is limited to the last four major versions of Android. For example, when a new, major version of Android is released, the Android requirement is the new version and the three most recent versions that precede it.

Currently there's one available Video Effect: Background Blur.

The VideoEffectsLocalVideoStreamFeature object has the following API structure:

enableEffect: Enables a Video Effect on the LocalVideoStream instance.
disableEffect: Disables a Video Effect on the LocalVideoStream instance.
OnVideoEffectEnabledListener: Event that is triggered when a Video Effect has been enabled successfully.
OnVideoEffectDisabledListener: Event that is triggered when a Video Effect has been disabled successfully.
OnVideoEffectErrorListener: Event that is triggered when a Video Effect operation fails.

The VideoEffectEnabledEvent, VideoEffectDisabledEvent and VideoEffectErrorEvent objects have the following API structure:

getVideoEffectName: Gets the name of the Video Effect that triggered the event.

Once you have the VideoEffectsLocalVideoStreamFeature object, you can subscribe to the events:

// Obtain the Video Effects feature from the LocalVideoStream object that is sending the video.
VideoEffectsLocalVideoStreamFeature videoEffectsFeature = currentVideoStream.feature(Features.LOCAL_VIDEO_EFFECTS);

// Create event handlers for the events
private void handleOnVideoEffectEnabled(VideoEffectEnabledEvent args) {
}
private void handleOnVideoEffectDisabled(VideoEffectDisabledEvent args) {
}
private void handleOnVideoEffectError(VideoEffectErrorEvent args) {
}
 
// Subscribe to the events
videoEffectsFeature.addOnVideoEffectEnabledListener(this::handleOnVideoEffectEnabled);
videoEffectsFeature.addOnVideoEffectDisabledListener(this::handleOnVideoEffectDisabled);
videoEffectsFeature.addOnVideoEffectErrorListener(this::handleOnVideoEffectError);

and start using the APIs to enable and disable Video Effects:

videoEffectsFeature.enableEffect( {{VIDEO_EFFECT_TO_DISABLE}} );
videoEffectsFeature.disableEffect( {{VIDEO_EFFECT_TO_DISABLE}} );

Background blur

Background Blur is a Video Effect that allows a person's background to be blurred. In order to use Background Video Effect, you need to obtain a VideoEffectsLocalVideoStreamFeature feature from a valid LocalVideoStream.

To enable Background Blur Video Effect:

Create a method that obtains the VideoEFfects Feature subscribes to the events:

private void handleOnVideoEffectEnabled(VideoEffectEnabledEvent args) {
   Log.i("VideoEfects", "Effect enabled for effect " + args.getVideoEffectName());
}
private void handleOnVideoEffectDisabled(VideoEffectDisabledEvent args) {
   Log.i("VideoEfects", "Effect disabled for effect " + args.getVideoEffectName());
}
private void handleOnVideoEffectError(VideoEffectErrorEvent args) {
   Log.i("VideoEfects", "Error " + args.getCode() + ":" + args.getMessage()
           + " for effect " + args.getVideoEffectName());
}

VideoEffectsLocalVideoStreamFeature videoEffectsFeature;
public void createVideoEffectsFeature() {
    videoEffectsFeature = currentVideoStream.feature(Features.LOCAL_VIDEO_EFFECTS);
    videoEffectsFeature.addOnVideoEffectEnabledListener(this::handleOnVideoEffectEnabled);
    videoEffectsFeature.addOnVideoEffectDisabledListener(this::handleOnVideoEffectDisabled);
    videoEffectsFeature.addOnVideoEffectErrorListener(this::handleOnVideoEffectError);
}

Create a new Background Blur Video Effect object:

BackgroundBlurEffect backgroundBlurVideoEffect = new BackgroundBlurEffect();

Call EnableEffect on the videoEffectsFeature object:

public void enableBackgroundBlur() {
    videoEffectsFeature.enableEffect(backgroundBlurEffect);
}

To disable Background Blur Video Effect:

public void disableBackgroundBlur() {
    videoEffectsFeature.disableEffect(backgroundBlurEffect);
}

Background replacement

Background Replacement is a Video Effect that allows a person's background to be replaced. In order to use Background Video Effect, you need to obtain a VideoEffectsLocalVideoStreamFeature feature from a valid LocalVideoStream.

To enable Background Replacement Video Effect:

Create a method that obtains the VideoEFfects Feature subscribes to the events:

private void handleOnVideoEffectEnabled(VideoEffectEnabledEvent args) {
   Log.i("VideoEfects", "Effect enabled for effect " + args.getVideoEffectName());
}
private void handleOnVideoEffectDisabled(VideoEffectDisabledEvent args) {
   Log.i("VideoEfects", "Effect disabled for effect " + args.getVideoEffectName());
}
private void handleOnVideoEffectError(VideoEffectErrorEvent args) {
   Log.i("VideoEfects", "Error " + args.getCode() + ":" + args.getMessage()
           + " for effect " + args.getVideoEffectName());
}

VideoEffectsLocalVideoStreamFeature videoEffectsFeature;
public void createVideoEffectsFeature() {
    videoEffectsFeature = currentVideoStream.feature(Features.LOCAL_VIDEO_EFFECTS);
    videoEffectsFeature.addOnVideoEffectEnabledListener(this::handleOnVideoEffectEnabled);
    videoEffectsFeature.addOnVideoEffectDisabledListener(this::handleOnVideoEffectDisabled);
    videoEffectsFeature.addOnVideoEffectErrorListener(this::handleOnVideoEffectError);
}

Create a new Background Replacement Video Effect object:

BackgroundReplacementEffect backgroundReplacementVideoEffect = new BackgroundReplacementEffect();

Set a custom background by passing in the image through a buffer.

//example of where we can get an image from, in this case, this is from assets in Android folder
InputStream inputStream = getAssets().open("image.jpg");
Bitmap bitmap = BitmapFactory.decodeStream(inputStream);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.JPEG, 100, stream);
byte[] data = stream.toByteArray();
ByteBuffer dataBuffer = ByteBuffer.allocateDirect(data.length);
dataBuffer.put(data);
dataBuffer.rewind();
backgroundReplacementVideoEffect.setBuffer(dataBuffer);

Call EnableEffect on the videoEffectsFeature object:

public void enableBackgroundReplacement() {
    videoEffectsFeature.enableEffect(backgroundReplacementVideoEffect);
}

To disable Background Replacement Video Effect:

public void disableBackgroundReplacement() {
    videoEffectsFeature.disableEffect(backgroundReplacementVideoEffect);
}

Note

In order to use Video Effects on the Windows Calling SDK, a machine learning model is downloaded to the customer's device. We encourage you to review the privacy notes in your application and update them accordingly, if necessary.

This quickstart builds on Quickstart: Add 1:1 video calling to your app for Windows.

Using video effects

Currently there's one available Video Effect: Background Blur.

The VideoEffectsLocalVideoStreamFeature object has the following API structure:

EnableEffect: Enables a Video Effect on the LocalVideoStream instance.
DisableEffect: Disables a Video Effect on the LocalVideoStream instance.
VideoEffectEnabled: Event that is triggered when a Video Effect has been enabled successfully.
VideoEffectDisabledListener: Event that is triggered when a Video Effect has been disabled successfully.
VideoEffectErrorListener: Event that is triggered when a Video Effect operation fails.

The VideoEffectEnabledEvent, VideoEffectDisabledEvent and VideoEffectErrorEvent objects have the following API structure:

VideoEffectName: Gets the name of the Video Effect that triggered the event.

Once you have the VideoEffectsLocalVideoStreamFeature object, you can subscribe to the events:

To use Video Effects with the Azure Communication Calling SDK, add the variables to MainPage.

public sealed partial class MainPage : Page
{
    private LocalVideoEffectsFeature localVideoEffectsFeature;
}

Once you've created a LocalVideoStream, you need to get the VideoEffects feature API of the LocalVideoStream to enable/disable Video Effects.

private async void CameraList_SelectionChanged(object sender, SelectionChangedEventArgs e)
{
    var selectedCamerea = CameraList.SelectedItem as VideoDeviceDetails;
    cameraStream = new LocalOutgoingVideoStream(selectedCamerea);
    InitVideoEffectsFeature(cameraStream);
    
    var localUri = await cameraStream.StartPreviewAsync();
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
    {
        LocalVideo.Source = MediaSource.CreateFromUri(localUri);
    });
}

public void InitVideoEffectsFeature(LocalOutgoingVideoStream videoStream){
    localVideoEffectsFeature = videoStream.Features.VideoEffects;
    localVideoEffectsFeature.VideoEffectEnabled += LocalVideoEffectsFeature_VideoEffectEnabled;
    localVideoEffectsFeature.VideoEffectDisabled += LocalVideoEffectsFeature_VideoEffectDisabled;
    localVideoEffectsFeature.VideoEffectError += LocalVideoEffectsFeature_VideoEffectError;
}

// Create event handlers for the events
private void LocalVideoEffectsFeature_VideoEffectEnabled(object sender, VideoEffectEnabledEventArgs args)
{
}

private void LocalVideoEffectsFeature_VideoEffectDisabled(object sender, VideoEffectDisabledEventArgs args)
{
}

private void LocalVideoEffectsFeature_VideoEffectError(object sender, VideoEffectErrorEventArgs args)
{
}
 
// Subscribe to the events
videoEffectsFeature.VideoEffectEnabled += VideoEffectsFeature_OnVideoEffectEnabled;
videoEffectsFeature.VideoEffectDisabled += VideoEffectsFeature_OnVideoEffectDisabled;
videoEffectsFeature.VideoEffectError += VideoEffectsFeature_OnVideoEffectError;

and start using the APIs to enable and disable Video Effects:

videoEffectsLocalVideoStreamFeature.EnableEffect( {{VIDEO_EFFECT_TO_ENABLE}} );
videoEffectsLocalVideoStreamFeature.DisableEffect( {{VIDEO_EFFECT_TO_DISABLE}} );

Background blur

To enable Background Blur Video Effect:

Add the BackgroundBlurEffect instance to the MainPage.

public sealed partial class MainPage : Page
{
    private BackgroundBlurEffect backgroundBlurVideoEffect = new BackgroundBlurEffect();
}

Create a method that obtains the VideoEFfects Feature subscribes to the events:

private async void LocalVideoEffectsFeature_VideoEffectEnabled(object sender, VideoEffectEnabledEventArgs e)
{
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, async () =>
    {
        BackgroundBlur.IsChecked = true;
    });
}

private async void LocalVideoEffectsFeature_VideoEffectDisabled(object sender, VideoEffectDisabledEventArgs e)
{
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, async () =>
    {
        BackgroundBlur.IsChecked = false;
    });
}

private void LocalVideoEffectsFeature_VideoEffectError(object sender, VideoEffectErrorEventArgs e)
{
    String effectName = args.VideoEffectName;
    String errorCode = args.Code;
    String errorMessage = args.Message;

    Trace.WriteLine("VideoEffects VideoEffectError on effect " + effectName + "with code "
        + errorCode + "and error message " + errorMessage);
}

Enable and disable the Background Blur effect:

private async void BackgroundBlur_Click(object sender, RoutedEventArgs e)
{
    if (localVideoEffectsFeature.IsEffectSupported(backgroundBlurVideoEffect))
    {
        var backgroundBlurCheckbox = sender as CheckBox;
        if (backgroundBlurCheckbox.IsChecked.Value)
        {
            localVideoEffectsFeature.EnableEffect(backgroundBlurVideoEffect);
        }
        else
        {
            localVideoEffectsFeature.DisableEffect(backgroundBlurVideoEffect);
        }
    }
}

Background replacement

To enable Background Replacement Video Effect:

Add the BackgroundReplacementEffect instance to the MainPage.

public sealed partial class MainPage : Page
{
    private BackgroundReplacementEffect backgroundReplacementVideoEffect = new BackgroundReplacementEffect();
}

Create a method that obtains the VideoEFfects Feature subscribes to the events:

private async void LocalVideoEffectsFeature_VideoEffectEnabled(object sender, VideoEffectEnabledEventArgs e)
{
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, async () =>
    {
        BackgroundReplacement.IsChecked = true;
    });
}

private async void LocalVideoEffectsFeature_VideoEffectDisabled(object sender, VideoEffectDisabledEventArgs e)
{
    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, async () =>
    {
        BackgroundReplacement.IsChecked = false;
    });
}

private void LocalVideoEffectsFeature_VideoEffectError(object sender, VideoEffectErrorEventArgs e)
{
    String effectName = args.VideoEffectName;
    String errorCode = args.Code;
    String errorMessage = args.Message;

    Trace.WriteLine("VideoEffects VideoEffectError on effect " + effectName + "with code "
        + errorCode + "and error message " + errorMessage);
}

Set a custom background by passing in the image through a buffer.

//example of getting the image from storage folder
MemoryBuffer memoryBuffer = new MemoryBuffer(0);
StorageFolder InstallationFolder = Windows.ApplicationModel.Package.Current.InstalledLocation;
StorageFile file = InstallationFolder.GetFileAsync("image.jpg").GetAwaiter().GetResult();
if (File.Exists(file.Path))
{
    byte[] imageBytes = File.ReadAllBytes(file.Path);
    memoryBuffer = new MemoryBuffer((uint)imageBytes.Length);
    using (IMemoryBufferReference reference = memoryBuffer.CreateReference())
    {
        byte* dataInBytes;
        uint capacityInBytes;

        (reference.As<IMemoryBufferByteAccess>()).GetBuffer(out dataInBytes, out capacityInBytes);
        for (int i = 0; i < imageBytes.Length; i++)
        {
            dataInBytes[i] = imageBytes[i];
        }
    }
    return memoryBuffer;
}
backgroundReplacementVideoEffect.Buffer = memoryBuffer;

Enable and disable the Background Replacement effect:

private async void BackgroundReplacement_Click(object sender, RoutedEventArgs e)
{
    if (localVideoEffectsFeature.IsEffectSupported(backgroundReplacementVideoEffect))
    {
        var backgroundReplacementCheckbox = sender as CheckBox;
        if (backgroundReplacementCheckbox.IsChecked.Value)
        {
            localVideoEffectsFeature.EnableEffect(backgroundReplacementVideoEffect);
        }
        else
        {
            localVideoEffectsFeature.DisableEffect(backgroundReplacementVideoEffect);
        }
    }
}

Next steps

For more information, see the following articles:

Check out our calling hero sample
Get started with the UI Library
Learn about Calling SDK capabilities
Learn more about how calling works

QuickStart: Add video effects to your video calls

Using video effects

Install the package

Background blur

Background replacement with an image

Using video effects

Background blur

Background Replacement

Using video effects

Background blur

Background replacement

Using video effects

Background blur

Background replacement

Next steps

Tilbakemeldinger

Flere ressurser