Hi, I am looking for a way to stream video (without audio) to HoloLens 2 with a latency < 300ms? I'd like to create a resizable virtual window on the HoloLens that shows the video streams. So far I tried: "HTTP Live Streaming" (HLS) => latency too high WebRTC => not supported in Edge and Firefox Reality browser MP4 Progressive Download ("Pseudo-Streaming") => latency too high RTSP + RTP (over UDP) 4a. with VLC Player app => latency >1s even with :network-cache=0 etc. 4b. with Huczeq's "RTSP Player" app => best latency, but still approx 700ms My RTSP/RTP server (gst-rtsp-server) is already properly configured for near zero latency. I get < 150ms latency when streaming to gstreamer's rtspsrc module on another computer over wifi. Does the HoloLens 2 provide another way to receive and show a video stream with low latency? I haven't tested yet whether HoloLens' browser supports the Media Source Extension API (MSE).

Hello, Welcome to Microsoft Q&A, MixedReality-WebRTC can help you to integrate peer-to-peer real-time video communication into your application, which can be used outside Web Browser. It is recommended you start with this tutorial to create a simple unity application to receive remote media. For more information please refer to: MixedReality-WebRTC ---- If the response is helpful, please click " Accept Answer " and upvote it. Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

How to stream video to HoloLens 2 with low latency?

Fabian 6

Hi,

I am looking for a way to stream video (without audio) to HoloLens 2 with a latency < 300ms?
I'd like to create a resizable virtual window on the HoloLens that shows the video streams.

So far I tried:

"HTTP Live Streaming" (HLS) => latency too high
WebRTC => not supported in Edge and Firefox Reality browser
MP4 Progressive Download ("Pseudo-Streaming") => latency too high
RTSP + RTP (over UDP)
4a. with VLC Player app => latency >1s even with :network-cache=0 etc.
4b. with Huczeq's "RTSP Player" app => best latency, but still approx 700ms

My RTSP/RTP server (gst-rtsp-server) is already properly configured for near zero latency.
I get < 150ms latency when streaming to gstreamer's rtspsrc module on another computer over wifi.

Does the HoloLens 2 provide another way to receive and show a video stream with low latency?

I haven't tested yet whether HoloLens' browser supports the Media Source Extension API (MSE).

marryshah000987 0 Reputation points

2024-06-09T17:35:37.27+00:00

Hi Fabian,

It sounds like you've tried several methods to stream video to the HoloLens 2 with low latency. Have you considered using the Media Source Extension (MSE) API? It might help reduce latency further if supported by the HoloLens browser.

Also, while not directly related, ensure any other applications like Lightroom APK aren't consuming excessive resources that could impact streaming performance.

Best of luck!

Best,
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Fabian 6 Reputation points

2024-06-15T21:02:31.7933333+00:00

Hi @marryshah000987, my previous (more detailed) reply was deleted by Microsoft. In summary: We abandoned our plans to stream to the Hololens with low latency. Thank you!
Aldoha 0 Reputation points

2024-06-20T08:25:45.15+00:00

My RTSP/RTP server (gst-rtsp-server) is already

1 answer

Hernando Ren 2,171 Reputation points

2021-10-07T08:32:00.953+00:00

Hello, Welcome to Microsoft Q&A,

MixedReality-WebRTC can help you to integrate peer-to-peer real-time video communication into your application, which can be used outside Web Browser. It is recommended you start with this tutorial to create a simple unity application to receive remote media. For more information please refer to:MixedReality-WebRTC

----
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
Please sign in to rate this answer.

1 person found this answer helpful.
Fabian 6 Reputation points

2021-10-09T14:16:46.947+00:00

Thank you! I'll check it out

_rom-b 1 Reputation point

2021-10-12T13:27:47.947+00:00

Hi Fabian,

could you share your results with WebRTC when you tried it? Would be much appreciated.
Cheers
Roman

Fabian 6 Reputation points

2021-10-12T13:34:40.99+00:00

Hi, sure! I tried to run the "Hello, C# world!" example today, but apparently the library's internal dependency libmrwebrtc.so can not be built on Linux. I'll setup a Windows virtual machine and try again there

Fabian 6 Reputation points

2022-06-27T18:19:01.043+00:00

Hi @_rom-b , after some testing we decided against using https://github.com/microsoft/MixedReality-WebRTC cause the build system with windows virtual machines was quite cumbersome and the repository was archived / marked as deprecated

Roman Bibo 1 Reputation point

2022-06-28T07:36:20.637+00:00

Hey @Fabian . Thanks for the info! Actually, we tried MRWebRTC with Unity on Hololens 2 and a windows PC. It works for now, but it has its limits. For instance frame rates drop when you try to send more than 2 or 3 video tracks at once.
We haven't done any performance tests, yet. But as you already mentioned, the repo is deprecated and they already removed it from MixedRealityFeatureTool. So, it doesn't seem to be the right way to do it.

Do you have any other recommendations?

Fabian 6 Reputation points

2022-06-28T18:19:34.64+00:00

Hey, thanks a lot for your update / report! Unfortunately, we haven't found any alternatives with acceptable latency yet

Santiago 0 Reputation points

2024-05-20T16:37:47.4266667+00:00

Hello @Roman Bibo , I am trying to receive a WebRTC stream from the HoloLens using a WebRTC server and WHEP. Do you know if that is possible? I see that they used node-dss, which is not quite what I need. How did you manage to send multiple video sources? Do you have a GitHub repository or something? Thank you!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to stream video to HoloLens 2 with low latency?

1 answer

Your answer