MS Teams | Automation API & Accessibility Insight | Dual Monitor

Question

Issue
Accessibility behavior of MS Teams on Dual Monitor, with Monitors setup at different scales, my specific example 100% and 125%, with 1920*1080 resolution (or with different combinations of Scaling, Resolution and Physical dimensions of the Monitors in Dual/Multi Monitor setups) do not seem to follow any deterministic behavior. Consequently, element detection is incorrect. I have tried to check the result from Automation API and Accessibility Insight and they both show similar behavior with unproportionate offsets.

From Task Manager, I see MS Teams Process, responsible for UI, is a Per Monitor DPI Aware and so is Accessibility Insight, I also have developed a prototype using Windows Automation API that runs on Per Monitor DPI aware mode. However, both the Prototype (and Accessibility Insight) are unable to locate the MS Teams' Elements correctly in dual Monitor, different scale configuration. The behavior is non-deterministic and is clearly visible with below settings.

Further, MS Teams responds well on single Monitor with any scale factor or resolution for elements that I have tested.

I have raised an issue with Accessibility Insight Team here and on stackoverflow . In response to Stackoverflow question, Accessibility Team has mentioned that the issue seems be with MS Teams Application. After requesting for information on MS Community and Github and in absence of an answer, I raise it here.

To Reproduce
Steps to reproduce the behavior:

1. Use Dual Monitors with Monitor 1 as Primary. Monitor 1 is set at 100%, 1920x1080 while Monitor 2 is set at 125%, 1920x1080. Both of Monitors are 14 inches which keeps Physical Screen Size out of equation.
2. Launch Teams App in Monitor 1 (100%) and move it to Monitor 2 (125%)
3. Open Accessibility Insight in Monitor 1 (or Monitor 2)
4. Try to locate "Search" box on Title Bar of Teams, where you can Search for a Person. You can try any other Element as well.
5. The Bounding Rectangle I see has Left as 2935 and Top as 280 while the actual Physical Coordinates that I get using GetPhysicalCursorPos (user32.dll) is X:2454 & Y:10
6. I wrote a Prototype that hosts Windows Automation API (code snippet below) and ran it in DPI Unaware, System DPI Aware and Per Monitor DPI Aware mode. In DPI Unaware Mode and System DPI Aware Mode, I get Bounding Rectangle Left as 2348 and Top as 10. If I scale the Bounding Rectangle Left value, obtained in step 5, down by 1.25 to account for Scaling (2935/1.25), I get 2348. Which makes me believe that scaling is done on presuming the Monitor is on 100%, probably because it was launched on Monitor 1 with 100% scale factor. Considering left location of Mouse is at 2454 and Top is at 10, I find it difficult to establish relation here.
7. When I run the Prototype in Per Monitor DPI Aware mode, I get value Bounding Rectangle Left and Top as 2935 and 280, that matches with Accessibility Insight.
8. I have also tried 14 inch and 22 inch monitors' combinations with 100% and 125% respectively and 1920*1080 resolutions. The result is left seems to be aligned however Top still gives an offset of 280.

Expected behavior
Considering Teams app runs in Per Monitor DPI Aware mode and both the Monitors are 14 inches, I would expect the Prototype (and Accessibility Insight) to give the correct location of the Element/s. In this case it should be Left ~2454 and Top ~10, actual Physical position of the cursor obtained from GetPhysicalCursorPos. Further, I understand that behavior would depend on Physical Screen size, Resolution and Scale Factor, however, I cannot find a correlation that I could rely on for all possible combinations.

Actual behavior
Teams App doesn't seem to respond correctly to Monitor Scaling.
Screenshots or .GIF

Location of Search box pointed by Mouse (top left of the box)

DPI Awareness of Teams Process

Per Monitor DPI Aware Response
image

DPI Unaware response

Value of left 2935 in Per Monitor DPI Aware mode is scaled up version of the value of Cursor position in DPI Unaware mode, 2347. However, actual Cursor position is 2454 in Per Monitor DPI Aware mode.

Prototype hosting Automation API

        private static void DpiScaleIssue(IntPtr handle)  
        {  
            var cp = new CursorPoint();  
            GetPhysicalCursorPos(ref cp);  
            Console.WriteLine("Physical Cursor Position X:{0} Y:{1}", cp.X, cp.Y);  
            var flag = PhysicalToLogicalPointForPerMonitorDPI(IntPtr.Zero, ref cp);  
  
            if (flag)  
                Console.WriteLine("After applying PhysicalToLogicalPointForPerMonitorDPI X:{0} Y:{1}", cp.X, cp.Y);  
  
            GetPhysicalCursorPos(ref cp);  
            flag = PhysicalToLogicalPoint(IntPtr.Zero, ref cp);  
  
            if (flag)  
                Console.WriteLine("After applying PhysicalToLogicalPoint X:{0} Y:{1}", cp.X, cp.Y);  
  
  
            GetPhysicalCursorPos(ref cp);  
            flag = LogicalToPhysicalPointForPerMonitorDPI(IntPtr.Zero, ref cp);  
  
            if (flag)  
                Console.WriteLine("After applying LogicalToPhysicalPointForPerMonitorDPI X:{0} Y:{1}", cp.X, cp.Y);  
  
            GetPhysicalCursorPos(ref cp);  
            flag = LogicalToPhysicalPoint(IntPtr.Zero, ref cp);  
  
            if (flag)  
                Console.WriteLine("After applying LogicalToPhysicalPoint X:{0} Y:{1}", cp.X, cp.Y);  
  
  
            Console.WriteLine();  
            Console.WriteLine("***Automation API***");  
            var conditions = new IUIAutomationCondition[1];  
            conditions[0] = _automation.CreatePropertyCondition(UIA_PropertyIds.UIA_ControlTypePropertyId, 50004);  
            //conditions[1] = _automation.CreatePropertyCondition(UIA_PropertyIds.UIA_NamePropertyId, "new 4");  
            while (true)  
            {  
                var element = _automationRootElement.FindFirst(TreeScope.TreeScope_Descendants, _automation.CreateAndConditionFromArray(conditions));  
                Console.WriteLine("Found element {0}", element.CurrentName);  
                Console.WriteLine("Automation API Left: {0}, Top: {1}", element.CurrentBoundingRectangle.left, element.CurrentBoundingRectangle.top);  
                //MoveWindow(handle, 1920, 1030, 1921, 1031, false);  
                cp.X = element.CurrentBoundingRectangle.left;  
                cp.Y = element.CurrentBoundingRectangle.top;  
  
                flag = PhysicalToLogicalPoint(handle, ref cp);  
                if (flag)  
                    Console.WriteLine("After applying PhysicalToLogicalPoint X:{0} Y:{1}", cp.X, cp.Y);  
  
                cp.X = element.CurrentBoundingRectangle.left;  
                cp.Y = element.CurrentBoundingRectangle.top;  
  
                flag = PhysicalToLogicalPointForPerMonitorDPI(handle, ref cp);  
                if (flag)  
                    Console.WriteLine("After applying PhysicalToLogicalPointForPerMonitorDPI X:{0} Y:{1}", cp.X, cp.Y);  
  
                cp.X = element.CurrentBoundingRectangle.left;  
                cp.Y = element.CurrentBoundingRectangle.top;  
                flag = LogicalToPhysicalPoint(handle, ref cp);  
                if (flag)  
                    Console.WriteLine("After applying LogicalToPhysical X:{0} Y:{1}", cp.X, cp.Y);  
  
                cp.X = element.CurrentBoundingRectangle.left;  
                cp.Y = element.CurrentBoundingRectangle.top;  
                flag = LogicalToPhysicalPointForPerMonitorDPI(handle, ref cp);  
                if (flag)  
                    Console.WriteLine("After applying LogicalToPhysicalPointForPerMonitorDPI X:{0} Y:{1}", cp.X, cp.Y);  
  
  
                Console.ReadLine();  
            }  
        }  
  

System Details
OS: [e.g. Windows 10 1909]
Accessibility Insights for Windows Version:
Target Application: [MS Teams]
Target Application Version: [e.g. 1.4.00.8872]

Additional context
If I enable GDI scaling, reading Improve High DPIExperience, I do see that Prototype as well as Accessibility Insight is able to locate the Element as it should which indicates that Per Monitor DPI Awareness doesn't function as it should. Further, Accessibility Insight works well on "Display Settings" itself (SystemSettings.exe process), which is also Per Monitor Aware.