Edit

Creating a spatial awareness system data provider — MRTK2

  • Article

The Spatial Awareness system is an extensible system for providing applications with data about real world environments. To add support for a new hardware platform or a new form of Spatial Awareness data, a custom data provider may be required.

This article describes how to create custom data providers, also called Spatial Observers, for the Spatial Awareness system. The example code shown here is from the SpatialObjectMeshObserver class implementation which is useful for loading 3D mesh data in-editor.

Note

The complete source code used in this example can be found in the Assets/MRTK/Providers/ObjectMeshObserver folder.

Namespace and folder structure

Data providers can be distributed in one of two ways:

  1. Third party add-ons
  2. Part of Microsoft Mixed Reality Toolkit

The approval process for submissions of new data providers to MRTK will vary on a case-by-case basis and will be communicated at the time of the initial proposal. Proposals can be submitted by creating a new Feature Request type issue.

Third party add-on

Namespace

Data providers are required to have a namespace to mitigate potential name collisions. It is recommended that the namespace includes the following components.

  • Company name producing the add-on
  • Feature area

For example, a Spatial Awareness data provider created and shipped by the Contoso company may be "Contoso.MixedReality.Toolkit.SpatialAwareness".

Folder structure

It is recommended that the source code for data providers be layed out in a folder hierarchy as shown in the following image.

Example folder structure

Where the ContosoSpatialAwareness folder contains the implementation of the data provider, the Editor folder contains the inspector (and any other Unity editor specific code), and the Profiles folder contains one or more pre-made profile scriptable objects.

MRTK submission

Namespace

If a spatial awareness system data provider is being submitted to the Mixed Reality Toolkit repository, the namespace must begin with Microsoft.MixedReality.Toolkit (ex: Microsoft.MixedReality.Toolkit.SpatialObjectMeshObserver)

and the code should be located in a folder beneath MRTK/Providers (ex: MRTK/Providers/ObjectMeshObserver).

Folder structure

All code should be located in a folder beneath MRTK/Providers (ex: MRTK/Providers/ObjectMeshObserver).

Define the spatial data object

The first step in creating a Spatial Awareness data provider is determining the type of data (ex: meshes or planes) it will provide to applications.

All spatial data objects must implement the IMixedRealitySpatialAwarenessObject interface.

The Mixed Reality Toolkit foundation provides the following spatial objects that can be used or extended in new data providers.

Implement the data provider

Specify interface and/or base class inheritance

All Spatial Awareness data providers must implement the IMixedRealitySpatialAwarenessObserver interface, which specifies the minimum functionality required by the Spatial Awareness system. The MRTK foundation includes the BaseSpatialObserver class which provides a default implementation of this required functionality.

public class SpatialObjectMeshObserver :
    BaseSpatialObserver,
    IMixedRealitySpatialAwarenessMeshObserver,
    IMixedRealityCapabilityCheck
{ }

Note

The IMixedRealityCapabilityCheck interface is used by the SpatialObjectMeshObserver class to indicate that it provides support for the SpatialAwarenessMesh capability.

Apply the MixedRealityDataProvider attribute

A key step in creating a Spatial Awareness data provider is to apply the MixedRealityDataProvider attribute to the class. This step enables setting the default profile and platform(s) for the data provider, when selected in the Spatial Awareness profile as well as Name, folder path, and more.

[MixedRealityDataProvider(
    typeof(IMixedRealitySpatialAwarenessSystem),
    SupportedPlatforms.WindowsEditor | SupportedPlatforms.MacEditor | SupportedPlatforms.LinuxEditor,
    "Spatial Object Mesh Observer",
    "ObjectMeshObserver/Profiles/DefaultObjectMeshObserverProfile.asset",
    "MixedRealityToolkit.Providers")]
public class SpatialObjectMeshObserver :
    BaseSpatialObserver,
    IMixedRealitySpatialAwarenessMeshObserver,
    IMixedRealityCapabilityCheck
{ }

Implement the IMixedRealityDataProvider methods

Once the class has been defined, the next step is to provide the implementation of the IMixedRealityDataProvider interface.

Note

The BaseSpatialObserver class, via the BaseService class, provides only an empty implementations for IMixedRealityDataProvider methods. The details of these methods are generally data provider specific.

The methods that should be implemented by the data provider are:

  • Destroy()
  • Disable()
  • Enable()
  • Initialize()
  • Reset()
  • Update()

Implement the data provider logic

The next step is to add the logic of the data provider by implementing the specific data provider interface, for example IMixedRealitySpatialAwarenessMeshObserver. This portion of the data provider will typically be platform specific.

Observation change notifications

To allow applications to respond to changes in the device's understanding of the environment, the data provider raises notification events as defined in the IMixedRealitySpatialAwarenessObservationtHandler<T> interface.

  • OnObservationAdded()
  • OnObservationRemoved()
  • OnObservationUpdated()

The following code from the SpatialObjectMeshObserver examples demonstrates raising and event when mesh data is added.

// The data to be sent when mesh observation events occur.
// This member variable is initialized as part of the Initialize() method.
private MixedRealitySpatialAwarenessEventData<SpatialAwarenessMeshObject> meshEventData = null;

/// <summary>
/// Sends the observations using the mesh data contained within the configured 3D model.
/// </summary>
private void SendMeshObjects()
{
    if (!sendObservations) { return; }

    if (spatialMeshObject != null)
    {
        MeshFilter[] meshFilters = spatialMeshObject.GetComponentsInChildren<MeshFilter>();
        for (int i = 0; i < meshFilters.Length; i++)
        {
            SpatialAwarenessMeshObject meshObject = SpatialAwarenessMeshObject.Create(
                meshFilters[i].sharedMesh,
                MeshPhysicsLayer,
                $"Spatial Object Mesh {currentMeshId}",
                currentMeshId,
                ObservedObjectParent);

            meshObject.GameObject.transform.localPosition = meshFilters[i].transform.position;
            meshObject.GameObject.transform.localRotation = meshFilters[i].transform.rotation;

            ApplyMeshMaterial(meshObject);

            meshes.Add(currentMeshId, meshObject);

            // Initialize the meshEventData variable with data for the added event.
            meshEventData.Initialize(this, currentMeshId, meshObject);
            // Raise the event via the spatial awareness system.
            SpatialAwarenessSystem?.HandleEvent(meshEventData, OnMeshAdded);

            currentMeshId++;
        }
    }

    sendObservations = false;
}

Note

The SpatialObjectMeshObserver class does not raise OnObservationUpdated events since the 3D model is only loaded once. The implementation in the WindowsMixedRealitySpatialMeshObserver class provides an example of raising an OnObservationUpdated event for an observed mesh.

Add Unity Profiler instrumentation

Performance is critical in mixed reality applications. Every component adds some amount of overhead for which applications must account. To this end, it is important that all spatial awareness data providers contain Unity Profiler instrumentation in inner loop and frequently utilized code paths.

It is recommended to implement the pattern utilized by MRTK when instrumenting custom providers.

        private static readonly ProfilerMarker UpdateObserverPerfMarker = new ProfilerMarker("[MRTK] WindowsMixedRealitySpatialMeshObserver.UpdateObserver");

        /// <summary>
        /// Requests updates from the surface observer.
        /// </summary>
        private void UpdateObserver()
        {
            using (UpdateObserverPerfMarker.Auto())
            {
                // Code to be measured.
            }
        }

Note

The name used to identify the profiler marker is arbitrary. MRTK uses the following pattern.

"[product] className.methodName - optional note"

It is recommended that custom data providers follow a similar pattern to help simplify identification of specific components and methods when analyzing traces.

Create the profile and inspector

In Mixed Reality Toolkit, data providers are configured using profiles.

Define the profile

Profile contents should mirror the accessible properties of the data provider (ex: update interval). All of the user configurable properties defined in each interface should be contained with the profile.

Base classes are encouraged if a new data provider extends an existing provider. For example, the SpatialObjectMeshObserverProfile extends the MixedRealitySpatialAwarenessMeshObserverProfile to enable customers to provide a 3D model to be used as the environment data.

[CreateAssetMenu(
    menuName = "Mixed Reality Toolkit/Profiles/Spatial Object Mesh Observer Profile",
    fileName = "SpatialObjectMeshObserverProfile",
    order = 100)]
public class SpatialObjectMeshObserverProfile : MixedRealitySpatialAwarenessMeshObserverProfile
{
    [SerializeField]
    [Tooltip("The model containing the desired mesh data.")]
    private GameObject spatialMeshObject = null;

    /// <summary>
    /// The model containing the desired mesh data.
    /// </summary>
    public GameObject SpatialMeshObject => spatialMeshObject;
}

The CreateAssetMenu attribute can be applied to the profile class to enable customers to create a profile instance using the Create > Assets > Mixed Reality Toolkit > Profiles menu.

Implement the inspector

Profile inspectors are the user interface for configuring and viewing profile contents. Each profile inspector should extend the BaseMixedRealityToolkitConfigurationProfileInspector class.

The CustomEditor attribute informs Unity the type of asset to which the inspector applies.

[CustomEditor(typeof(SpatialObjectMeshObserverProfile))]
public class SpatialObjectMeshObserverProfileInspector : BaseMixedRealityToolkitConfigurationProfileInspector
{ }

Create assembly definition(s)

Mixed Reality Toolkit uses assembly definition (.asmdef) files to specify dependencies between components as well as to assist Unity in reducing compilation time.

It is recommended that assembly definition files are created for all data providers and their editor components.

Using the folder structure in the earlier example, there would be two .asmdef files for the ContosoSpatialAwareness data provider.

The first assembly definition is for the data provider. For this example, it will be called ContosoSpatialAwareness and will be located in the example's ContosoSpatialAwareness folder. This assembly definition must specify a dependency on Microsoft.MixedReality.Toolkit and any other assemblies upon which it depends.

The ContosoInputEditor assembly definition will specify the profile inspector and any editor specific code. This file must be located in the root folder of the editor code. In this example, the file will be located in the ContosoSpatialAwareness\Editor folder. This assembly definition will contain a reference to the ContosoSpatialAwareness assembly as well as:

  • Microsoft.MixedReality.Toolkit
  • Microsoft.MixedReality.Toolkit.Editor.Inspectors
  • Microsoft.MixedReality.Toolkit.Editor.Utilities

Register the data provider

Once created, the data provider can be registered with the Spatial Awareness system to be used in the application.

Selecting the spatial object mesh observer

Packaging and distribution

Data providers that are distributed as third party components have the specific details of packaging and distribution left to the preference of the developer. Likely, the most common solution will be to generate a .unitypackage and distribute via the Unity Asset Store.

If a data provider is submitted and accepted as a part of the Microsoft Mixed Reality Toolkit package, the Microsoft MRTK team will package and distribute it as part of the MRTK offerings.

See also