Does HL2 indeed not support vergence distance/point?

Question

Using the ExtendedEyeGazeDataProvider provided by the extended eye tracking capability of MRTK3, GazeReading.VergenceDistanceSupported returns false when run on HL2. Based on that, I can only assume that this property is indeed not supported on HL2. Can anyone confirm? If so, are there any other ideas on extracting the vergence point/distance on the HL2?

What came to my mind was to determine it manually by combining the single eye rays and estimating a vergence point via their potential intersection.

Ultimate goal is to extract some 2D gaze coordinates in screen space of a camera based on user gaze. These coordinates are needed even if gaze does not intersect with virtual holograms does making the approach of raycasting against objects infeasible.

In that regard I am currently also testing to raycast against the spatial mesh. However, the concern here is that the mesh is coarse and imprecise, leading to wrong gaze coordinates in the end.

Answer 1

Yes, your observation is correct. The HoloLens 2 does not directly provide a vergence distance/point calculation. Here's a breakdown of the situation and potential workarounds:

Why vergence distance is likely not supported:

Hardware Limitations: Unlike some advanced VR/AR headsets, the HoloLens 2 might not have the precise eye-tracking sensors to accurately calculate the convergence point of the eyes.

Focus on Holograms: The HoloLens 2 is primarily designed for interacting with holographic content. Direct vergence distance may be less of a priority, especially if the primary use case is focusing on objects in the user's environment.

Alternative approaches:

1. Estimating Vergence Point: Your idea of manually intersecting eye gaze rays is a viable approach. Here's what you'd need to consider:

• Accuracy: The accuracy depends on the precision of your eye-tracking data. You might need to implement some smoothing or calibration to reduce noise.
• Computation: Real-time ray intersection calculations can add some overhead. Optimize if performance is critical.

1. Spatial Mesh Raycasting: Your point about the coarse mesh is valid. Here are some ways to address that:

• Refinement: Explore options to increase the spatial mesh resolution or create a custom, higher-detail invisible mesh purely for eye-tracking.
• Interpolation: If you hit the spatial mesh, use surrounding points to refine the gaze position estimate through interpolation.

1. Hybrid Approach: Combine the strengths of both:

Use spatial mesh raycasting as a general guide to determine where the user is roughly looking.

If the mesh hit is imprecise, fall back to your ray intersection method for finer gaze estimation.

Extracting 2D Gaze Coordinates:

Once you have a vergence point (real or estimated) here's how to obtain 2D screen space coordinates:

1. Camera Model: Know your camera's intrinsic parameters (focal length, principal point, etc.)
2. Head Pose: Get the HoloLens 2's position and orientation in world space.
3. Projection: Project the vergence point using the camera model and head pose onto the image plane to get screen space coordinates.

Libraries/Tools

• MRTK: The Mixed Reality Toolkit likely has helper functions for camera projections and raycasting.
• OpenCV: OpenCV's computer vision capabilities could be useful for eye-tracking calibration or refining spatial mesh data.

Important Note: Eye-tracking on the HoloLens 2, particularly for precise gaze coordinate calculation, is still an area of active research and development. You might have to experiment to find the best solution for your needs.