Eye-supported navigation in MRTK — MRTK2

MRTK

Imagine you are reading information on a slate and when you reach the end of the displayed text, the text automatically scrolls up to reveal more content. Or you can fluently zoom in where you are looking at. The map also automatically adjusts the content to keep the things of interest within your field of view. Another interesting application is the hands-free observation of 3D holograms by automatically bringing the parts of the hologram that you are looking at to the front. These are some of the examples that are described on this page in context of eye-supported navigation.

The following descriptions assume that you are already familiar with how to set up eye tracking in your MRTK scene and with the basics of accessing eye tracking data in MRTK Unity. The examples discussed in the following are all part of the EyeTrackingDemo-03-Navigation (Assets/MRTK/Examples/Demos/EyeTracking/Scenes/EyeTrackingDemo-03-Navigation) scene.

Summary: Auto scroll of text, eye-gaze-supported pan and zoom of a virtual map, hands-free gaze-directed 3D rotation.

Auto scroll

Auto scroll enables the user to scroll through texts without lifting a finger. Simply continue reading and the text will automatically scroll up or down depending on where the user is looking. You can start off from the example provided in EyeTrackingDemo-03-Navigation (Assets/MRTK/Examples/Demos/EyeTracking/Scenes). This example uses a TextMesh component to allow for flexibly loading and formatting new text. To enable auto scroll, simply add the following two scripts to your collider component of the textbox:

ScrollRectTransf

To scroll through a TextMesh or more generally speaking a RectTransform component you can use the ScrollRectTransf script. If you want to scroll through a texture instead of a RectTransform, use ScrollTexture instead of ScrollRectTransf. In the following, the parameters of ScrollRectTransf that are available in the Unity Editor are explained in more detail:

Parameters Description
LimitPanning If enabled, will stop the scrollable content at its boundary.
RectTransfToNavigate Reference to the RectTransform to scroll in.
RefToViewport Reference to the parent RectTransform of the scrollable content to determine the correct offset and boundary.
AutoGazeScrollIsActive If enabled, the text will automatically scroll if the user looks at an active region (e.g., the top and bottom part of your scroll panel if the vertical scroll speed is not zero).
ScrollSpeed_x If set to a value unequal to zero, horizontal scroll will be enabled. Negative values mean a change in scroll direction: Left to right vs. right to left.
ScrollSpeed_y If set to a value unequal to zero, vertical scroll will be enabled. Negative values mean a change in scroll direction: Up to down vs. down to up.
MinDistFromCenterForAutoScroll Normalized minimal distance in x and y from center of the target's hit box (0, 0) to scroll. Thus, values must range between 0 (always scroll) and 0.5 (no scroll).
UseSkimProofing If enabled, it prevents sudden scroll movements when quickly looking around. This may make scrolling feel less responsive though. It can be tuned with the SkimProofUpdateSpeed value.
SkimProofUpdateSpeed The lower the value, the slower the scrolling will speed up after skimming. Recommended value: 5.

Eye-supported scroll setup in Unity

EyeTrackingTarget

Attaching the EyeTrackingTarget component allows for flexibly handle eye-gaze-related events. The scroll sample demonstrates scrolling text that starts when the user looks at the panel and stops when the user is looking away from it. Eye-supported scroll setup in Unity: EyeTrackingTarget

Gaze-supported pan and zoom

Who hasn't used a virtual map before to search for their home or to explore entirely new places? Eye tracking allows you to directly dive into exactly the parts that you're interested in and once zoomed in, you can smoothly follow the course of a street to explore your neighborhood! This is not only useful for exploring geographical maps, but also to checking out details in photographs, data visualizations or even live-streamed medical imagery. To use this capability in your app is easy! For content rendered to a Texture (e.g., a photo, streamed data), simply add the PanZoomTexture script. For a RectTransform use PanZoomRectTransf. Extending the Auto Scroll capability, we essentially enable to scroll both vertically and horizontally at the same time and magnify content right around the user's current focus point.

Parameters Description
LimitPanning If enabled, will stop the scrollable content at its boundary.
HandZoomEnabledOnStartup Indicates whether hand gestures are automatically enabled to perform a zoom gesture. You may want to disable it at first to avoid accidentally triggering zoom actions.
RendererOfTextureToBeNavigated Referenced renderer of the texture to be navigated.
Zoom_Acceleration Zoom acceleration defining the steepness of logistic speed function mapping.
Zoom_SpeedMax Maximum zoom speed.
Zoom_MinScale Minimum scale of the texture for zoom in - e.g., 0.5f (half the original size).
Zoom_MaxScale Maximum scale of the texture for zoom out - e.g., 1f (the original size) or 2.0f (double the original size).
Zoom_TimeInSecToZoom Timed zoom: Once triggered, a zoom in/out will be performed for the given amount of time in seconds.
Zoom_Gesture Type of hand gesture to use to zoom in/out.
--- ---
Pan_AutoScrollIsActive If enabled, the text will automatically scroll if the user looks at an active region (e.g., the top and bottom part of your scroll panel if the vertical scroll speed is not zero).
Pan_Speed_x If set to a value unequal to zero, horizontal scroll will be enabled. Negative values mean a change in scroll direction: Left to right vs. right to left.
Pan_Speed_y If set to a value unequal to zero, vertical scroll will be enabled. Negative values mean a change in scroll direction: Up to down vs. down to up.
Pan_MinDistFromCenter Normalized minimal distance in x and y from center of the target's hit box (0, 0) to scroll. Thus, values must range between 0 (always scroll) and 0.5 (no scroll).
UseSkimProofing If enabled, it prevents sudden scroll movements when quickly looking around. This may make scrolling feel less responsive though. It can be tuned with the SkimProofUpdateSpeed value.
SkimProofUpdateSpeed The lower the value, the slower the scrolling will speed up after skimming. Recommended value: 5.

Eye-supported pan and zoom setup in Unity

Attention-based 3D rotation

Imagine looking at a 3D object and the parts you want to see more closely magically turn toward you - as if the system would read your mind and know to turn the item toward you! That is the idea for attention-based 3D rotations which enable you to investigate all side of a hologram without lifting a finger. To enable this behavior, simply add the OnLookAtRotateByEyeGaze script to the part of your GameObject with a Collider component. You can tweak several parameters that are listed below to limit how fast and in which directions the hologram will turn.

As you can imagine, having this behavior active at all times may quickly become pretty distracting in a crowded scene. This is why you may want to start out with this behavior disabled and then enable it quickly using voice commands. Alternatively, we added an example in EyeTrackingDemo-03-Navigation (Assets/MRTK/Examples/Demos/EyeTracking/Scenes) to use TargetMoveToCamera for which you can select a focused target and it flies in front of you - simply say "Come to me".

Once in the near mode, the auto rotation mode is automatically enabled. In that mode, you can observe it from all sides either simply leaning back and looking at it, walking around it or reaching out to grab and rotate it with your hand. When you dismiss the target (look & pinch or say "Send back"), it will return to its original location and will stop reacting to you from afar.

Parameters Description
SpeedX Horizontal rotation speed.
SpeedY Vertical rotation speed.
InverseX To inverse the horizontal rotation direction.
InverseY To inverse the vertical rotation direction.
RotationThreshInDegrees If the angle between 'Gaze to Target' and 'Camera to Target' is less than this value, do nothing. This is to prevent small jittery rotations..
MinRotX Minimum horizontal rotation angle. This is to limit the rotation in different directions.
MaxRotX Maximum horizontal rotation angle. This is to limit the rotation in different directions.
MinRotY Minimal vertical rotation angle to limit the rotation around the x axis.
MaxRotY Maximum vertical rotation angle to limit the rotation around the y axis.

Eye-supported 3D rotation setup in Unity

In summary, the above scripts should enable you to get started to use eye gaze for various input navigation tasks such as scrolling texts, zooming and panning textures as well as rotating investigating 3D holograms.

See also


Back to "Eye tracking in the MixedRealityToolkit"