Get the ElementFromPoint (IUIAutomation) from the 2nd scaled monitor

Question

Hello,
I'm getting the wrong IUIAutomation element from the 2nd monitor, if use it in a browser window like edge (which is not the main monitor and has a dpi for i.e. 125% in the windows settings.)
I tried a few things and can't find a working solution.

The following is not working (they are all far above the element under the mouse cursor):

• GetCursorPos
• GetPhysicalCursorPos
• Mousehook (hookStruct.pt.X + " "+ hookStruct.pt.Y)
• PhysicalToLogicalPointForPerMonitorDPI (don't understand how i could get the cursor position since i'm getting the window's data?!)

I've read:
https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-and-screen-scaling
https://support.microsoft.com/en-us/topic/windows-scaling-issues-for-high-dpi-devices-508483cd-7c59-0d08-12b0-960b99aa347d

It's not supported by a method, so i have to calculate it by myself after getting GetCursorPos?! (Getting the dpi, the monitor bounds to offset it against the GetCursorPos.)

[...and while i'm asking, i'm reading that it could be working by SetThreadDpiAwarenessContext function? (because it looks really weird, if my form's dpiAwareness is true). ------ FAILED]

EDIT: I also tried to get the dpi and monitor bounds for calculating it by the following example (gives me really weird data) :

https://stackoverflow.com/questions/29438430/how-to-get-dpi-scale-for-all-screens

Isn't there a solution for just getting the set dpi from the windows settings for the specific monitor and simply calculating the right cursor position for finding the element?!

Regards

Answer 1

Ok,
tested (also with inspect.exe) and read the whole day/night and it doesn't work. Could it be that this is not supported by UIA (the different DPIs on multiple monitors for interacting with windows of other applications)?!
In my case:
I have 1 monitor and a notebook. The monitor is 100% and the notebook on 125%. It properly works, if i set my notebook to 100% but it hurts my eyes :).
I tried it by setting the dpiAwareness in a manifest file and all the methods but it makes no difference.
If i set the cursor tagPoint (x & y) for the ElementFromPoint by code, it also behaves totally unpredictable.
That's really unsatisfactory.

Answer 2

According to the attached documents, It‘s needed to make the client application dpi-aware firstly(Try SetProcessDpiAwareness(PROCESS_PER_MONITOR_DPI_AWARE)). And then GetPhysicalCursorPos because the UI Automation API does not use logical coordinates.
Perhaps you should have a try on Highlighter which is mentioned in the previous document.
I have found the Positioning Objects on a Multiple Display Setup sample.
Finally, we can convert between logical and physical coordinates by using the Win32 functions PhysicalToLogicalPoint and LogicalToPhysicalPoint in regard of non-DPIAwake process.

Answer 3

Hi XiaopoYang,
will try it again today evening after work. Are you sure that it must work with my settings (1 monitor & 1 laptop/ both 1920 * 1080/ the first on 100% and the laptop on 125% )?!
I've tried SetProcessDpiAwareness and all cominations and parameters (i also tried the manifest file because it's recommended in the documentation).

All methods i've tried as i can remember:

• GetPhysicalCursorPos (it's not different from GetCursorPos [logical position])
• LogicalToPhysicalPoint (not supported anymore)
• PhysicalToLogicalPointForPerMonitorDPI
• LogicalToPhysicalPointForPerMonitorDPI
• SetThreadDpiAwarenessContext

As i mentioned i tried it by setting the x and y by code which didn't change the behaviour (if i change it one by one, I can't even get to the cursor, it's so weird). I always get an other element far above the cursor on the 125% monitor and it's exactly the same behaviour like in inspect.exe that i don't want (it marks an an element far above and left).

public static void ShowUIAData(IUIAutomationElement element)
        {
            Console.WriteLine("------------------------------------------------");
            Console.WriteLine("ShowUIAData");
            Console.WriteLine("- NAME: " + element.CurrentName);
            Console.WriteLine("- ARIAROLE: " + element.CurrentAriaRole);
            Console.WriteLine("- CONTROLTYPE: " + (ControlType)element.CurrentControlType);
            Console.WriteLine("- ITEMTYPE: " + element.CurrentItemType);
            Console.WriteLine("- FOCUSABLE: " + element.CurrentIsKeyboardFocusable);
            Console.WriteLine("- CurrentName: " + element.CurrentName);
            Console.WriteLine("- CurrentClassName: " + element.CurrentClassName);
            Console.WriteLine("- CurrentControlType: " + element.CurrentControlType);
        }

 public static IUIAutomationElement ElementFromCursor()
    {
        var automation = new CUIAutomation();
        var point = new Point(Cursor.Position.X, Cursor.Position.Y);
        var tp = new tagPOINT
        {
            x = (int)point.X,
            y = (int)point.Y
        };

        return automation.ElementFromPoint(tp);
    }

Regards

Answer 4

Hi,
sorry for the late reply, was ill 1 week.
I think you don't understand the task for me here.
I need the element under the cursor independent of the monitor dpi.
My 2 monitors have 2 different dpi settings for that i can work properly on them.
UiAutomation doesn't give me the element under the cursor on the second monitor.
It doesn't work properly, if the second monitor has a different dpi than the main monitor.

Looked at your examples:

• OnFocusChanged (won't help me?! top level window/ focused element)
• Multiple Accessibility (can't find it under the link?!)
• I have also tried the managed Api and DPI Awareness again. I really don't understand how could that work because it doesn't.

I need the element under cursor because i have a function that depends on it.
The function checks the element under the cursor in for i.e. browsers (apps without handles).
Since websites also have for i.e. menus like for i.e. (i posted it month ago):

https://learn.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.runtimehelpers.preparemethod?redirectedfrom=MSDN&view=net-6.0#System_Runtime_CompilerServices_RuntimeHelpers_PrepareMethod_System_RuntimeMethodHandle_System_RuntimeTypeHandle___
I have to consider this and really have to get the element under the cursor because of further calculations.

Background:
I have to consider different ariaroles and controltypes which are not uniformly because of the webdesign conditions in general.
Then i have to get "the element under the cursor" to get the needed element closest to it (not in the for i.e. side menu) as fast as possible (takes to long for usability which i have to treat next).

Best regards

Answer 5

Ah, it works without enabling the dpi awareness with the controls of an other application (this is what you mean?!) but not on the content of a website in a browser...