Unreal and HoloLens 2 Research Mode - How?

Nico Zink 1 Reputation point
2021-05-14T11:09:40.993+00:00

Hello everyone!

I'm new here, so feel free to give advice even when not directly related to my question. I am running into trouble using the Unreal engine combined with the HoloLens 2.

I would like to access the special black/white cameras of the HoloLens, for tracking purposes. These are normally not accessible. However, they can be activated by using the “perceptionSensorsExperimental” capability. This should be possible, since it also works with Unity: https://github.com/doughtmw/HoloLensForCV-Unity

I have tried to add the capability in the Unreal Project Settings: Config\HoloLens\HoloLensEngine.ini” -> “+RescapCapabilityList=perceptionSensorsExperimental”. The project still builds as expected, but I noticed that it doesn’t matter what I add here. Even something random like “+abcd=efgh” doesn’t break the build. However, if I add “+CapabilityList=perceptionSensorsExperimental”, I get “Packaging (HoloLens): ERROR: The 'Name' attribute is invalid - The value 'perceptionSensorsExperimental' is invalid according to its datatype 'http://schemas.microsoft.com/appx/manifest/types:ST_Capability_Foundation' - The Enumeration constraint failed.”. I conclude: 1.) I’m making the changes in the right file. 2.) The right scheme needs to be configured in order for “+RescapCapabilityList=perceptionSensorsExperimental” to work as expected.

My question is how do I add the right schema to my Unreal project? (like in the Unity example referenced above, which uses “http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities”), I cannot find any example and I cannot find any proper place to put it. Not in the settings, not in the xml/ini files. Clearly, I am missing something.

Any thoughts are much appreciated!

HoloLens Development
HoloLens Development
HoloLens: A family of Microsoft self-contained, holographic devices that enable engagement with digital content and interaction with holograms in the surrounding environment.Development: The process of researching, productizing, and refining new or existing technologies.
379 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Andrey Chistyakov 6 Reputation points
    2021-09-30T14:28:21.417+00:00

    We have released an Unreal plugin for Hololens 2 Research Mode.

    https://github.com/microsoft/HoloLens-ResearchMode-Unreal

    It require some customisation in Unreal source code to get working.

    1 person found this answer helpful.
    0 comments
  2. Franklin Chen 1 Reputation point Microsoft Employee
    2021-05-17T01:38:35.143+00:00

    I don't see any possibility of Unreal has direct support for research mode capabilities. A possible way is to build Unreal engine from source code, you can modify the key method here. If you are not very familiar, this way may take a lot of time.

    The another possible way is to create a plugin that calls directly into the Research Mode libraries. You should then be able to add the plugin into Unreal and consume it via that path.