Training
Learning path
Use advance techniques in canvas apps to perform custom updates and optimization - Training
Use advance techniques in canvas apps to perform custom updates and optimization
App quality criteria overview
This document describes the top factors impacting the quality of mixed reality apps. For each factor, the following information is provided:
- Overview – a brief description of the quality factor and why it's important.
- Device impact - which type of Window Mixed Reality device is affected.
- Quality criteria – how to evaluate the quality factor.
- How to measure – methods to measure (or experience) the issue.
- Recommendations – summary of approaches to provide a better user experience.
- Resources – relevant developer and design resources that are useful to create better app experiences.
Frame rate is the first pillar of hologram stability and user comfort. Frame rate below the recommended targets can cause holograms to appear jittery, negatively impacting the believability of the experience and potentially causing eye fatigue. The target frame rate for your experience on Windows Mixed Reality immersive headsets is either 60 Hz or 90 Hz depending on which Windows Mixed Reality-compatible PCs you're supporting. For HoloLens, the target frame rate is 60 Hz.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Meets | Fail |
|---|---|---|
| The app consistently meets frames per second (FPS) goal for target device: 60 fps on HoloLens; 90 fps on Ultra PCs; and 60 fps on mainstream PCs. | The app has intermittent frame drops not impeding the core experience, or FPS is consistently lower than desired goal but doesn’t impede the app experience. | The app is experiencing a drop in frame rate on average every 10 seconds or less. |
- A real-time frame rate graph is provided by the Windows Device Portal under "System Performance."
- For development debugging, add a frame rate diagnostic counter into the app. See Resources for a sample counter.
- Frame rate drops can be experienced in device while the app is running by moving your head from side to side. If the hologram shows unexpected jittery movement, then low frame rate or the stability plane is likely the cause.
- Add a frame rate counter at the beginning of the development work.
- Changes that incur a drop in frame rate should be evaluated and appropriately resolved as a performance bug.
- Understanding Performance for Mixed Reality
- Hologram stability and framerate
- Asset performance budget
- Performance recommendations for Unity
Stable holograms will increase the usability and believability of your app and create a more comfortable viewing experience for the user. The quality of hologram stability is a result of good app development and the device's ability to understand (track) its environment. While frame rate is the first pillar of stability, other factors can impact stability, including:
- Use of the stabilization plane
- Distance to spatial anchors
- Tracking
| Best | Meets | Fail |
|---|---|---|
| Holograms consistently appear stable. | Secondary content shows unexpected movement; or unexpected movement doesn't impede overall app experience. | Primary content in frame shows unexpected movement. |
While wearing the device and viewing the experience:
- Move your head from side to side. If the holograms show unexpected movement, then low frame rate or improper alignment of the stability plane to the focal plane is the likely cause.
- Move around the holograms and environment and look for behaviors such as swim and jumpiness. This type of motion is likely caused by the device not tracking the environment, or the distance to the spatial anchor.
- If large or multiple holograms are in the frame, observe hologram behavior at various depths while moving your head position from side to side. If shakiness appears, it's likely caused by the stabilization plane.
- Add a frame rate counter at the beginning of the development work.
- Use the stabilization plane.
- Always render anchored holograms within three meters of their anchor.
- Make sure your environment is set up for proper tracking.
- Design your experience to avoid holograms at various focal depth levels within the frame.
- Hologram stability and framerate
- Understanding Performance for Mixed Reality
- Performance recommendations for Unity
- Spatial anchors
- Handling tracking errors
- Stationary frame of reference
Misalignments of holograms with physical objects (if intended to be placed in relation to one another) are a clear indication of the non-union of holograms and real-world. Accuracy of the placement should be relative to the needs of the scenario; for example, general surface placement can use the spatial map, but more accurate placement will require some use of markers and calibration.
| Best | Meets | Fail |
|---|---|---|
| Holograms align to the surface typically in the centimeters to inches range. If you need more accuracy, the app should provide an efficient means for collaboration within the app spec. | NA | The holograms appear unaligned with the physical target object by either breaking the surface plane or appearing to float away from the surface. If accuracy is required, Holograms should meet the proximity spec of the scenario. |
- Holograms that are placed on spatial map shouldn't appear to dramatically float above or below the surface.
- Holograms that require accurate placement should have some form of marker and calibration system that is accurate to the scenario's requirement.
- Spatial map is useful for placing objects on surfaces when precision isn’t required.
- For the best precision, use markers or posters to set the holograms and an Xbox controller (or some manual alignment mechanism) for final calibration.
- Consider breaking extra-large holograms into logical parts and aligning each part to the surface.
- Improperly set interpupillary distance (IPD) can also effect hologram alignment. Always configure HoloLens to the user's IPD.
- Spatial mapping placement
- Room scanning process
- Spatial anchors best practices
- Handling tracking errors
- Spatial mapping in Unity
- Vuforia development overview
- MR Toolkit, Spatial Mapping Libraries
- MR Companion Kit, Poster Calibration Sample
- MR Companion Kit, Kinect IPD
App developers control where users' eyes converge by placing content and holograms at various depths. Users wearing HoloLens will always accommodate to 2.0 m to maintain a clear image because HoloLens displays are fixed at an optical distance approximately 2.0 m away from the user. Improper content depth can lead to visual discomfort or fatigue.
| Best |
|
| Meets | Content is within the viewing and motion guidance, but improper use or no use of the clipping plane. |
| Fail | Content is presented too close (typically <1.25 m, or <50 cm for stationary holograms requiring closer observation.) |
- Content should typically be 2 m away, but no closer than 1.25 or further than 5 m.
- With few exceptions, the HoloLens clipping render distance should be set to 85CM with fade out of content starting at 1 m. Approach the content and note the clipping plane effect.
- Stationary content should not be closer than 50 cm away.
- Design content for the optimal viewing distance of 2 m.
- Set the clipping render distance to 85 cm with fade out of content starting at 1 m.
- For stationary holograms that need closer viewing, the clipping plane should be no closer than 30 cm and fade out should start at least 10 cm away from the clipping plane.
- Comfort with HoloLens
- Render distance
- Focus point in Unity
- Experimenting with scale
- Text, Recommended font size
Regardless of viewing zone of comfort issues, demands for the user to switch frequently or quickly between near and far focal objects (including holograms and real-world content) can lead to oculomotor fatigue, and general discomfort.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Meets | Fail |
|---|---|---|
| Limited or natural depth switching that doesn’t cause the user to unnaturally refocus. | Abrupt depth switch this is core and designed into the app experience, or abrupt depth switch that is caused by unexpected real-world content. | Consistent depth switch, or abrupt depth switching that isn’t necessary or core to the app experience. |
- If the app requires the user to consistently and/or abruptly change depth focus, there is depth switching problem.
- Keep primary content at a consistent focal plane and make sure the stabilization plane matches the focal plane. This will alleviate oculomotor fatigue and unexpected hologram movement.
In Windows Mixed Reality, the audio engine provides the aural component of the mixed reality experience by simulating 3D sound using direction, distance, and environmental simulations. Using spatial sound in an application allows developers to convincingly place sounds in a 3-dimensional space (sphere) all around the user. Those sounds will then seem as if they were coming from real physical objects or the mixed reality holograms in the user's surroundings. Spatial sound is a powerful tool for immersion, accessibility, and UX design in mixed reality applications.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Meets | Fail |
|---|---|---|
| Sound is logically spatialized, and the UX appropriately uses sound to assist with object discovery and user feedback. Sound is natural and relevant to objects and normalized across the scenario. | Spatial audio is used appropriately for believability but missing as means to help with user feedback and discoverability. | Sound is not spatialized as expected, and/or lack of sound to assist user within the UX. Or spatial audio was not considered or used in the design of the scenario. |
- In general, relevant sounds should emit from target holograms (eg., bark sound coming from holographic dog.)
- Sound cues should be used throughout the UX to assist the user with feedback or awareness of actions outside the holographic frame.
- Use spatial audio to assist with object discovery and user interfaces.
- Real sounds work better than synthesize or unnatural sound.
- Most sounds should be spatialized.
- Avoid invisible emitters.
- Avoid spatial masking.
- Normalize all sounds.
- Spatial sound
- Spatial sound design
- Spatial sound in Unity
- Case study, Spatial sound for HoloTour
- Case study, Using spatial sound in RoboRaid
Well-designed user experiences can create and maintain useful context of the virtual environment that extends around the users. Mitigating the effect of the FOV boundaries involves a thoughtful design of content scale and context, use of spatial audio, guidance systems, and the user's position. If done right, the user will feel less impaired by the FOV boundaries while having a comfortable app experience.
| Best | Meets | Fail |
|---|---|---|
| User never loses context and viewing is comfortable. Context assistance is provided for large objects. Discoverability and viewing guidance is provided for objects outside the frame. In general, motion design and scale of the holograms are appropriate for a comfortable viewing experience. | User never loses context, but extra neck motion may be required in limited situations. In limited situations scale causes holograms to break either the vertical or horizontal frame causing some neck motion to view holograms. | User likely to lose context and/or consistent neck motion is required to view holograms. No context guidance for large holographic objects, moving objects easy to lose outside the frame with no discoverability guidance, or tall holograms requires regular neck motion to view. |
- Context for a (large) hologram is lost or not understood due to being clipped at the boundaries.
- Locations of holograms are hard to find due to the lack of attention directors or content that rapidly moves in and out of the holographic frame.
- Scenario requires regular and repetitive up and down head motion to fully see a hologram resulting in neck fatigue.
- Start the experience with small objects that fit the FOV, then transition with visual cues to larger versions.
- Use spatial audio and attention directors to help the user find content that is outside the FOV.
- As much as possible, avoid holograms that vertically clip the FOV.
- Provide the user with in-app guidance for best viewing location.
- Holographic frame
- Case Study, HoloStudio UI and interaction design learnings
- Scale of objects and environments
- Cursors, Visual cues
Holograms should react to the user position in roughly the same ways that "real" objects do. A notable design consideration is UI elements that can't necessarily assume a user's position is stationary and adapt to the user's motion. Designing an app that correctly adapts to user position will create a more believable experience and make it easier to use.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Content and UI adapt to user positions allowing user to naturally interact with content within the scope of expected user movement. |
| Meets | UI adapts to the user position, but may impede the view of key content requiring the user to adjust their position. |
| Fail |
|
- All measurements should be done within a reasonable scope of the scenario. While user movement will vary, don’t try to trick the app with extreme user movement.
- For UI elements, relevant controls should be available regardless of user movement. For example, if the user is viewing and walking around a 3D map with zoom, the zoom control should be readily available to the user regardless of location.
- The user is the camera and they control the movement. Let them drive.
- Consider billboarding for text and menuing systems that would otherwise be world-locked or obscured if a user were to move around.
- Use tag-along for content that needs to follow the user while still allowing the user to see what is in front of them.
- Interaction design
- Color, light, and material
- Billboarding and tag-along
- Instinctual interactions
- Self-motion and user locomotion
Input interaction clarity is critical to an app's usability and includes input consistency, approachability, discoverability of interaction methods. User can use platform-wide common interactions without relearning. If the app has custom input, it should be clearly communicated and demonstrated.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Meets | Fail |
|---|---|---|
| Input interaction methods are consistent with Windows Mixed Reality provided guidance. Any custom input shouldn't be redundant with standard input (rather use standard interaction) and must be clearly communicated and demonstrated to the user. | Similar to best, but custom inputs are redundant with standard input methods. User can still achieve the goal and progress through the app experience. | Difficult to understand input method or button mapping. Input is heavily customized, doesn't support standard input, no instructions, or likely to cause fatigue and comfort issues. |
- The app uses consistent standard input methods.
- If the app has custom input, it's clearly communicated through:
- First-run experience
- Introductory screens
- Tooltips
- Hand coach
- Help section
- Voice over
- Use standard input methods whenever possible.
- Provide demonstrations, tutorials, and tooltips for non-standard input methods.
- Use a consistent interaction model throughout the app.
- Instinctual interactions
- Interactable objects
- Head-gaze and dwell
- Cursors
- Comfort and gaze
- Voice input
- Motion controllers
- Input porting guide for Unity
- Keyboard input in Unity
- Gaze in Unity
- Motion controllers in Unity
- Gestures in Unity
- Voice input in Unity
- Keyboard, mouse, and controller input in DirectX
- Head and eye gaze in DirectX
- Hands and motion controllers in DirectX
- Voice input in DirectX
- Case study: The pursuit of more personal computing
- Cast study: HoloStudio UI and interaction design learnings
- Sample app: Periodic table of the elements
- Sample app: Lunar module
A button has long been a metaphor used for triggering an event in the 2D abstract world. In the three-dimensional mixed reality world, we don’t have to be confined to this world of abstraction anymore. Anything can be an Interactable object that triggers an event. An interactable object can be represented as anything from a coffee cup on the table to a balloon floating in the air. Regardless of the form, interactable objects should be clearly recognizable by the user through visual and audio cues.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Meets | Fail |
|---|---|---|
| Regardless of form, interactable objects are recognizable through visual and audio cues across three states: idle, targeted, and selected. "See it, say it" is clear and consistently used throughout the experience. Objects are scaled and distributed to allow for error free targeting. | User can recognize object as interactable through audio or visual feedback, and can target and activate the object. | Given no visual or audio cues, user can't recognize an interactable object. Interactions are error prone due to object scale or distance between objects. |
- Interactable objects are recognizable as 'interactable'; including buttons, menus, and app-specific content. As a rule of thumb there should be a visual and audio cue when targeting interactable objects.
- Use visual and audio feedback for interactions.
- Visual feedback should be differentiated for each input state (idle, targeted, selected)
- Interactable objects should be scaled and placed for error free targeting.
- Grouped interactable objects (such as a menu bar or list) should have proper spacing for targeting.
- Buttons and menus that support voice command should provide text labels for the command keyword ("See it, say it")
Apps that require spatial mapping data rely on the device to automatically collect this data over time and across sessions as the user explores their environment with the device active. The completeness and quality of this data depends on a number of factors including the amount of exploration the user has done, how much time has passed since the exploration and whether objects such as furniture and doors have moved since the device scanned the area. Many apps will analyze the spatial mapping data at the start of the experience to judge whether the user should perform additional steps to improve the completeness and quality of the spatial map. If the user is required to scan the environment, clear guidance should be provided during the scanning experience.
| Best | Meets | Fail |
|---|---|---|
| Visualization of the spatial mesh tell users scanning is in progress. User clearly knows what to do and when the scan starts and stops. | Visualization of the spatial mesh is provided, but the user may not clearly know what to do and no progress information is provided. | No visualization of mesh. No guidance information provided to the user regarding where to look, or when the scan starts/stops. |
- During a required room scan, visual and audio guidance are provided indicating where to look, and when to start and stop scanning.
- Indicate how much of the total volume in the users vicinity needs to be part of the experience.
- Communicate when the scan starts and stops such as a progress indicator.
- Use a visualization of the mesh during the scan.
- Provide visual and audio cues to encourage the user to look and move around the room.
- Inform the user where to go to improve the data. In many cases, it may be best to tell the user what they need to do (e.g. look at the ceiling, look behind furniture), in order to get the necessary scan quality.
- Room scan visualization
- Case study: Expanding the spatial mapping capabilities of HoloLens
- Case study: Spatial sound design for HoloTour
- Case study: Creating an immersive experience in Fragments
In a mixed reality app, content may be outside the field of view or occluded by real-world objects. A well-designed app will make it easier for the user to find non-visible content. Directional indicators alert a user to important content and provide guidance to the content relative to the user's position. Guidance to non-visible content can take the form of sound emitters, directional arrows, or direct visual cues.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Meets | Fail |
|---|---|---|
| Visual and audio cues directly guide the user to relevant content outside the field of view. | An arrow or some indicator that points the user in the general direction of the content. | Relevant content is outside of the field of view, and poor or no location guidance is provided to the user. |
- Relevant content outside of the user field of view is discoverable through visual and/or audio cues.
- When relevant content is outside the user's field of view, use directional indicators and audio cues to guide the user to the content. In many cases, a direct visual guide is preferred over directional arrows.
- Directional indicators should not be built into the cursor.
A progress control provides feedback to the user that a long-running operation is underway. It can mean that the user can't interact with the app when the progress indicator is visible and can also indicate how long the wait time might be.
| HoloLens | Immersive headsets | |
| ✔️ | ✔️ |
| Best | Meets | Fail |
|---|---|---|
| Animated visual indicator, in the form of a progress bar or ring, showing progress during any data loading or processing. The visual indicator provides guidance on how long the wait could be. | User is informed that data loading is in progress, but there is no indication of how long the wait could be. | No data loading or process indicators for task taking longer than 5 seconds. |
- During data loading verify there is no blank state for more than 5 seconds.
- Provide a data loading animator showing progress in any situation when the user may perceive this app to have stalled or crashed. A reasonable rule of thumb is any 'loading' activity that could take more than 5 seconds.
Additional resources
Documentation
-
Understanding Performance for Mixed Reality - Mixed Reality
Learn advanced information and details for analyzing and optimizing Windows Mixed Reality app performance.
-
Enable placement of 3D models in the home - Mixed Reality
Learn how to place 3D models from your website or application in the Windows Mixed Reality home.
-
Native development overview - Mixed Reality
Learn how to build a DirectX-based mixed-reality engine using the Windows Mixed Reality APIs directly.