ACS Calling SDK (JS): Stopping RawMedia stream with transformations causes Simulcast issue

Question

ACS Calling SDK (JS): Stopping RawMedia stream with transformations causes Simulcast issue

Joaquín Paz 20

Hi,

We're working on a video calls application using ACS Calling SDK in Typescript. We are using RawMedia to apply custom transformations to the stream. I'll try to be as detailed as possible regarding the issue, but just for context, the setup was working correctly, and we can start/stop streams with no issue (if not applying transformations).

We start the call with this.call.startVideo(LocalVideoStream) The local video stream is created with new LocalVideoStream(camera), camera being type VideoDeviceInfo

Everything works perfectly at this point. We proceed to transform the stream, pipe it through a transformer and generate a new stream. We then set this stream with LocalVideoStream.setMediaStream(transformedStream) and everything works perfectly.

We then proceed to this.call.stopVideo(LocalVideoStream) and that's when we get a Simulcast error in the console:

Uncaught (in promise) TypeError: Cannot read properties of null (reading 'mediaType')
    at SimulcastMediaStreamClient.get...

For reference: everything works perfectly, the video does stop, and there are no apparent issues other than the log. Can we please get help with this issue? Do we know why this error throws? Should we be doing something different when the stream was set with setMediaStream to stop the video?

Thanks for the help!

Kind regards,

Joaquin Paz

2 answers

Your answer

Answer 1

brtrach-MSFT 17,741 Microsoft Employee Moderator

@Joaquín Paz Thank you for reaching out to us. Based on the information you provided, it seems that the issue is related to the SimulcastMediaStreamClient. The error message indicates that it is trying to read properties of null, which means that the mediaType property is null. This could be caused by a few different things, such as a bug in the SDK or an issue with the configuration of the service.

To help us better understand the issue, could you please provide us with more information about the transformations you are applying to the stream? Also, could you please share the version of the ACS Calling SDK you are using?

In the meantime, you can try the following steps to see if they resolve the issue:

Make sure that you are using the latest version of the ACS Calling SDK.
Try stopping the video stream before setting the new transformed stream with LocalVideoStream.setMediaStream(transformedStream).
Check if the issue persists when you remove the transformer and use the original stream.

Joaquín Paz 20 Reputation points

2023-10-12T07:56:43.6633333+00:00
Thank you for the quick response @brtrach-MSFT

The version we are using for the ACS Calling SDK is 1.18.1-beta.1.

Regarding the steps, I've tried stopping the video stream before using LocalVideoStream.setMediaStream(transformedStream) and it still happens, this doesn't solve it. The issue also happens if I set a regular stream instead of a transformed one, so the transformations are not the issue.

My best bet would be to guess LocalVideoStream.setMediaStream either doesn't set mediaType correctly or changes some internal state or config that causes call.stopVideo(LocalVideoStream) to lose sync with Simulcast.

Below is a code snippet to recreate the issue:

const localVideoStream = new LocalVideoStream(camera) const stream = await localVideoStream.getMediaStream() await localVideoStream.setMediaStream(stream) await this.call.startVideo(localVideoStream)

It's redundant to set the same stream to the already created LocalVideoStream, but it's the most concise way to test the issue is in the setMediaStream method and not the stream.

Sometimes I have to toggle video off and on a couple of times to get the error log and sometimes I have to switch camera sources, but it usually always appears if I used setMediaStream
brtrach-MSFT 17,741 Reputation points Microsoft Employee Moderator

2023-10-12T19:09:40.4+00:00

@Joaquín Paz Since this sounds like a potential issue within the SDK, we would like to work closer with you on this matter. Can you please email us at azcommunity@microsoft.com with the subject as ATTN: Bryan and in the body of the email, please include your Azure Subscription ID. We look forward to hearing from you.
Joaquín Paz 20 Reputation points

2023-10-17T13:45:36.9533333+00:00

Just sent the email!
brtrach-MSFT 17,741 Reputation points Microsoft Employee Moderator

2023-10-20T00:50:53.63+00:00

Thanks Joquin!

Answer 2

Joaquín Paz 20

As a followup to this issue, aside from the log (that persists) we are having occasional bugs when ending calls where the camera remains in use (the stream doesn't close correctly). We are stopping the call with the LocalVideoStream call.stopVideo(localVideoStream)

Probably this issue could be locking/throwing the stream cleanup process. This bug usually happens when we start a lot of different calls in various tabs and stop them. Usually the first tabs close correctly, but after a couple the camera starts failing to release from the tabs (remaining in a locked state with the light on and the red-dot indicating it's in use by the browser).

Enrico-MSFT 5 Reputation points Microsoft Employee

2023-11-20T20:59:33.1933333+00:00
@Joaquín Paz for the following error:

Uncaught (in promise) TypeError: Cannot read properties of null (reading 'mediaType') at SimulcastMediaStreamClient.get...

We are able to reproduce the error with raw stream video. The error occurs during the stopVideo process when raw stream is being used. We have contacted the relevant team to look into it.

However, we are unable to reproduce the issue related to stream resources not being freed after the call ends. Could you describe more about the scenario? For example, is there any cloned media stream in the app that contains the active audio or video tracks after the call?
Joaquín Paz 20 Reputation points

2023-11-27T16:57:21.7066667+00:00

Thanks Enrico for the reply and apologies for the delay. I'm investigating on our end to see if any potential cloning is happening. I don't believe we are cloning streams, but I'm double checking to see if we're not closing it correctly once we create the custom transformed stream. I'll get back to you asap! Thanks

Joaquín Paz 20

After extensive research, we are not cloning streams, just setting them with localVideoStream.setMediaStream

This is the function we use to create a transformed video track and attach it to the media stream:

function makeInsertableStream(
  sourceVideoTrack: MediaStreamVideoTrack,
  transformer: Transformer,
  targetStreamToBeAttached: MediaStream,
  signal?: AbortSignal
) {
  const processor = new MediaStreamTrackProcessor({
    track: sourceVideoTrack,
  })
  const { readable: processorReadable } = processor

  const generator = new global.MediaStreamTrackGenerator({ kind: "video" })
  const { writable: generatorWritable } = generator

  const transform = new TransformStream(transformer)
  processorReadable
    .pipeThrough(transform, { signal })
    .pipeTo(generatorWritable, { signal })
    .catch(() => {
      // Avoid error propagation on stream stop
    })

  targetStreamToBeAttached.addTrack(generator)
  return targetStreamToBeAttached
}

Joaquín Paz 20 Reputation points

2023-12-14T02:30:48.8266667+00:00

We upgraded the sdk to 1.20.1-beta.2 and we got a more helpful log now: "Uncaught (in promise) webRtcSession cleanUp" and "Uncaught (in promise) 61", this last one stack traced to call.hangUp.

Does this shed some light on a possible cause?
Enrico-MSFT 5 Reputation points Microsoft Employee

2023-12-16T04:10:33.59+00:00

@Joaquín Paz for the Uncaught (in promise) TypeError: Cannot read properties of null (reading 'mediaType') error when stopVideo on the custom localVideoStream, we have found the root cause and it will be fixed in the upcoming beta release. (1.21.1-beta).

Regarding the stream release issue, what is the stream source you use for the sourceVideoTrack when you call makeInsertableStream? Does the sourceVideoTrack stop after the call is hung up if it's a video stream returned from the getUserMedia API?

Share via

ACS Calling SDK (JS): Stopping RawMedia stream with transformations causes Simulcast issue

2 answers

Your answer