Setting VideoStreamRenderer/LocalVideoStream max resolution (locally, not transmit resolution)

Question

Setting VideoStreamRenderer/LocalVideoStream max resolution (locally, not transmit resolution)

Neethan Puvanendran 0

Hi!

I'm using the ACS Javascript SDK to connect to a Team meeting, and trying to scale down the user's local video stream (not the stream they're sending to ACS). On most browsers, the local video stream is not rendered in HD and is fine for a local preview. The optional BackgroundBlurEffect() from communication-calling-effects works fine. However, on Safari on Mac OSX, the local video is rendered in the highest quality possible it seems. Using the same webcamera (Logitech C925e), the video is rendered in full HD on a Mac Mini running Safari. This is fine in itself, but turning on the background blur effect (which is supported in Safari on OSX according to the documentation), the video becomes extremely laggy and is not transmitted properly at all.

I assume this is either due to the use of the Babylon.js library for doing background blur in Safari, or due to the library not optimized for such high resolution. Anywho, I would like to be able to request a specific resolution for the the local video, as you can do using getUserMedia() using native functions. Is this possible to do currently?

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-14T15:11:46.23+00:00

Thanks for the follow-up and update. Sure, kindly let us know how it goes. I'm also discussing on this internally.
Neethan Puvanendran 0 Reputation points

2023-12-18T07:23:45.5466667+00:00
Hi again AJ,

I've tested on Safari 17.2 and still having the same issue. Turning on background blur either:

Freezes the browser

Turns on the blur, but lowers the FPS to around 3-4 (from a previous 60)

Does nothing

It's usually the first or second option, but during my testing it the background blur change did nothing at all. This may be a side effect of something else.

Our solution so far is to just disable the option to turn on background blur on Safari for our users, but this is not an optimal solution. Do update me and see if the team can reproduce this or not :)

Running a Mac Mini with an i3. Using a ultra wide Philips 499P9H at 5120x1440 if that makes a difference.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-19T20:13:01.6566667+00:00

Thanks for sharing the update and additional info about the issue and things you have tried to isolate. I have relayed this feedback internally, and will get back, as soon as I hear from my engineering team.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-21T19:10:43.2666667+00:00

Neethan Puvanendran, For additional investigation, kindly share the call ID, as outlined in the doc section - Enable and access call logs.

Note: Kindly do not share any PII data on the public forum for your privacy and security.
Neethan Puvanendran 0 Reputation points

2024-01-02T07:40:51.0666667+00:00

Hi again, I could share a call ID but this happens when using the APIs to preview a webcamera. Regardless, here's a call ID used while testing this: 989cab52-53c5-413c-a6d8-9ed97de94239
Neethan Puvanendran 0 Reputation points

2024-01-02T07:41:45.9466667+00:00

989cab52-53c5-413c-a6d8-9ed97de94239 is the call ID used, however, this happens during previewing of the local webcamera even when I'm not in a call. Just so that is clear.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-02T19:27:11.3966667+00:00

Thanks for sharing the requesting info, I'm checking on this and will follow-up on this shortly.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-08T18:18:28.27+00:00

Apologies for the delayed response. Utilizing direct access to raw media could be a good approach, enabling the extraction of the raw stream to manipulate it according to your needs before rendering it. See this doc: Quickstart: Add raw media access to your app.

Just to clarify, does it function properly on other browsers, or is the issue isolated to Safari on mobile devices? I'll further follow-up soon.
Neethan Puvanendran 0 Reputation points

2024-01-09T08:49:06.46+00:00

No worries. I don't believe it would be very viable to manipulate the raw video stream and add a background blur effect myself, when the effect is already implemented in the communication-calling library.

The effect works fine in Chrome and Edge on Windows, as per the documentation for the effects library. The effects library only supports these two browsers on Windows. On OSX, Safari is supported, but this is where I'm having issues. Safari on iOS does not support background blur, nor does any browsers on Android.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-09T20:19:22.1333333+00:00

Neethan Puvanendran, Thanks for the follow-up and update. As I understand, it's a Safari iOS a bug.
I have also relayed the feedback to our product engineering team.

I have converted comments to answer.

For the engagement on this thread, if the answer helped (pointed, you in the right direction) > please click Accept Answer.
Neethan Puvanendran 0 Reputation points

2024-01-12T09:15:26.5066667+00:00

No, this is a bug on Safari OSX. Background blur is not supported at all on Safari iOS. I see now that the title of this question is misleading, as the real issue is the background blur. I wanted to attempt to force a specific video resolution to see if the issue would be fixed, as on all other supported browsers (even Chrome on OSX) the background blur works as expected.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-15T04:30:19.3466667+00:00

Apologies for any confusion. Yes, bug on Safari OSX. Thanks for the follow-up and update.

1 answer

Your answer

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-14T15:11:46.23+00:00

Thanks for the follow-up and update. Sure, kindly let us know how it goes. I'm also discussing on this internally.
Neethan Puvanendran 0 Reputation points

2023-12-18T07:23:45.5466667+00:00

Hi again AJ,

I've tested on Safari 17.2 and still having the same issue. Turning on background blur either:

Freezes the browser

Turns on the blur, but lowers the FPS to around 3-4 (from a previous 60)

Does nothing

It's usually the first or second option, but during my testing it the background blur change did nothing at all. This may be a side effect of something else.

Our solution so far is to just disable the option to turn on background blur on Safari for our users, but this is not an optimal solution. Do update me and see if the team can reproduce this or not :)

Running a Mac Mini with an i3. Using a ultra wide Philips 499P9H at 5120x1440 if that makes a difference.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-19T20:13:01.6566667+00:00

Thanks for sharing the update and additional info about the issue and things you have tried to isolate. I have relayed this feedback internally, and will get back, as soon as I hear from my engineering team.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-21T19:10:43.2666667+00:00

Neethan Puvanendran, For additional investigation, kindly share the call ID, as outlined in the doc section - Enable and access call logs.

Note: Kindly do not share any PII data on the public forum for your privacy and security.
Neethan Puvanendran 0 Reputation points

2024-01-02T07:40:51.0666667+00:00

Hi again, I could share a call ID but this happens when using the APIs to preview a webcamera. Regardless, here's a call ID used while testing this: 989cab52-53c5-413c-a6d8-9ed97de94239
Neethan Puvanendran 0 Reputation points

2024-01-02T07:41:45.9466667+00:00

989cab52-53c5-413c-a6d8-9ed97de94239 is the call ID used, however, this happens during previewing of the local webcamera even when I'm not in a call. Just so that is clear.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-02T19:27:11.3966667+00:00

Thanks for sharing the requesting info, I'm checking on this and will follow-up on this shortly.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-08T18:18:28.27+00:00

Apologies for the delayed response. Utilizing direct access to raw media could be a good approach, enabling the extraction of the raw stream to manipulate it according to your needs before rendering it. See this doc: Quickstart: Add raw media access to your app.

Just to clarify, does it function properly on other browsers, or is the issue isolated to Safari on mobile devices? I'll further follow-up soon.
Neethan Puvanendran 0 Reputation points

2024-01-09T08:49:06.46+00:00

No worries. I don't believe it would be very viable to manipulate the raw video stream and add a background blur effect myself, when the effect is already implemented in the communication-calling library.

The effect works fine in Chrome and Edge on Windows, as per the documentation for the effects library. The effects library only supports these two browsers on Windows. On OSX, Safari is supported, but this is where I'm having issues. Safari on iOS does not support background blur, nor does any browsers on Android.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-09T20:19:22.1333333+00:00

Neethan Puvanendran, Thanks for the follow-up and update. As I understand, it's a Safari iOS a bug.
I have also relayed the feedback to our product engineering team.

I have converted comments to answer.

For the engagement on this thread, if the answer helped (pointed, you in the right direction) > please click Accept Answer.
Neethan Puvanendran 0 Reputation points

2024-01-12T09:15:26.5066667+00:00

No, this is a bug on Safari OSX. Background blur is not supported at all on Safari iOS. I see now that the title of this question is misleading, as the real issue is the background blur. I wanted to attempt to force a specific video resolution to see if the issue would be fixed, as on all other supported browsers (even Chrome on OSX) the background blur works as expected.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2024-01-15T04:30:19.3466667+00:00

Apologies for any confusion. Yes, bug on Safari OSX. Thanks for the follow-up and update.

Answer 1

ajkuma 28,036 Microsoft Employee Moderator

@Neethan Puvanendran , Thanks for posting this question. While I’m checking on this internally, kindly take a look at these points:

The following table represents the set of supported browsers, which are currently available. We support the most recent three major versions of the browser (most recent three minor versions for Safari) unless otherwise indicated.

Reference:

Just to highlight, as mentioned in this doc, Azure Communication Services Web Calling SDK supports setting the maximum video resolution, framerate, or bitrate that a client sends. The sender video constraints are supported on Desktop browsers (Chrome, Edge, Firefox) and when using iOS Safari mobile browser or Android Chrome mobile browser.

Ref: ACS Calling Web (JavaScript) SDK - Release History

We have a similar discussion, kindly check this:
[Azure Communications Service] Local video stream does not work on Safari MacOs #24556

Kindly share the following info: Provide the SDK version, call ID, code snippet that you are leveraging.

Note: Kindly do not share any PII data on the public forum for your privacy and security.

Neethan Puvanendran 0 Reputation points

2023-12-12T08:56:35.5566667+00:00
Thanks for the quick response. I have not tested if the sending constraints would adjust the local camera as well, but I assume it won't; in addition, in the use case for our application, we have a custom configuration wizard that the user goes through that uses the ACS library to display a camera feed, but does not connect to a call. As I see in the SDK, to use the sending constraints an active call is required.

SDK version:

"@azure/communication-calling": "^1.19.1", "@azure/communication-calling-effects": "^1.0.1",

Call ID: I have been continously testing on the same Teams meeting call for a while now. I will try to test on a new, unique call and send over the call ID from ACS.

I have tested this on Safari 16.6, but I see now that the latest version of Safari is 17.2 (where they also have fixed the slow input device change bug, another bug I was going to report). I will test my code in 17.2 and see if the issue persists. :)

I am unable to share a complete code snippet, but I can possibly create a simple test with a local webcamera feed and a single button to enable/disable background blur.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-12T15:24:36.74+00:00

Thanks for the follow-up and update. Sure, kindly let us know how it goes.

Does Video Constraints API help? Note that this feature of ACS is currently in preview.
Neethan Puvanendran 0 Reputation points

2023-12-12T21:59:36.7933333+00:00

Unfortunately, the Video Constraints API requires an active call to use. Our pre-call configuration page uses the ACS SDK to render the local video and devices in a list, as well as toggle background blur, but does so without creating a call. However, I could test this on the "in call" page and see if it makes a difference. I will get back to you on Thursday :)

Share via

Setting VideoStreamRenderer/LocalVideoStream max resolution (locally, not transmit resolution)

1 answer

Your answer