How to get ElementFromPoint?

Question

How to get ElementFromPoint?

D.D.K-2637 966

Hi,

I'm using the following code to get the element:

automation->ElementFromPoint(pt, &element)

however the problem is that I'm using camtasia for full screen recording, then the result element always returns camtasia window...

User's image

Is there a way to get the element from point in a certain application?

D.D.K-2637 966

Hi, I've seen the following instructions from doc:

to find elements in its own user interface, you must make all UI Automation calls on a separate thread.

but, I have called it like this without success..

Did I use the wrong parameter?

//xyz
void test (){
//...

thread t([]()
							{
								POINT pt{200,300};
								CComPtr<IUIAutomation> app;
								
								auto hr = CoCreateInstance(__uuidof(CUIAutomation),
									NULL, CLSCTX_INPROC_SERVER, __uuidof(IUIAutomation),
									(void**)&app);
							
                                CComPtr<IUIAutomationElement> element;
								if (SUCCEEDED(app->ElementFromPoint(pt,&element)))
								{
									...
									element->get_CurrentName(&name);
									outputformatW(L"\nname: {}", (wchar_t*)name.m_str);
									
									
								}
							});
//...

}

RLWA32 49,536 Reputation points

2023-05-04T08:20:57.9166667+00:00

I'm not familiar with camtasia.

What does inspect.exe show as the active element in the automation tree when the mouse cursor is positioned where the code would call ElementFromPoint?

Have you tried using the WindowFromPoint function to obtain an HWND for use with ElementFromHandle?
Castorix31 90,681 Reputation points

2023-05-04T08:56:15.83+00:00

If there is a TopMost/Layered window, it is normal.

I use ScreenToGIF and Inspect returns its window too
D.D.K-2637 966 Reputation points

2023-05-04T09:51:47.23+00:00

Hi. it is the transparent camtasia window that obscures everything, this is really annoying.

I hope to have more functionality like :

CComPtr<IUIAutomationElementArray> elements; automation->ElementArrayFromPoint(pt,&elements);

I tried WindowFromPoint(), ChildWindowFromPoint(), RealChildWindowFromPoint() but it doesn't return detail element.
RLWA32 49,536 Reputation points

2023-05-04T11:53:12.53+00:00

You may have success using MSAA to obtain a window handle that you can use with ElementFromHandle.

I used the ScreenToGif application mentioned by @Castorix31 over an instance of Notepad on Win10 22H2. As was noted earlier inspect.exe returned the ScreenToGif window in the automation tree. However, when I switched inspect.exe to MSAA mode (Accessibility) it found Notepad's windows from the mouse position.

I don't know if this will be the case with camtasia but its worth a try.

Update: If you can obtain an IAccessible interface pointer you can call IUIAutomation::ElementFromIAccessible method which doesn't need an HWND.
RLWA32 49,536 Reputation points

2023-05-10T19:51:58.68+00:00

@D.D.K-2637 Any updates to this issue?
D.D.K-2637 966 Reputation points

2023-05-10T22:34:34.24+00:00

Hi @RLWA32 ,

I missed your previous comment.

I just tried MSSA mode now with inspect, it returned the element node level2 of the parent window but the element looking for is on level 5. The problem is that the received attributes of this element seem to be different.

so maybe if i can use this element and switch the state back to automation mode then look for child elements i think it is a solution for now.
D.D.K-2637 966 Reputation points

2023-05-11T04:33:31.4266667+00:00

Hi @RLWA32 ,

I tried the ElementFromIAccessible Method, it seem required for an id to the element. but I haven't been able to determine the focus element (under cursor) so I probably won't know this id.
RLWA32 49,536 Reputation points

2023-05-11T07:47:07.3+00:00

As a starting point try passing CHILDID_SELF to the method. What happens then?

Update:

If you have an IAccessible interface pointer you can also try calling QueryInterface for IOleWindow. If it succeeds you can obtain an HWND to be used with ElementFromHandle. A useful utilty function to simplify this is IUnknown_GetWindow function
D.D.K-2637 966 Reputation points

2023-05-11T09:39:21.88+00:00

Thanks you.

I've seen IAccessible appear on inspect but it's pretty new to me now.

I will try it tonight.
RLWA32 49,536 Reputation points

2023-05-11T10:10:46.6733333+00:00

Your starting point would be to call the AccessibleObjectFromPoint function.

D.D.K-2637 966

Hi @RLWA32 ,

it worked but the result point to the parent element instead focus element .

The code I use:

CComPtr<IAccessible> iacc; 
VARIANT varchild; 
varchild.vt = VT_I4; 
varchild.intVal = CHILDID_SELF; 
AccessibleObjectFromPoint(m->pt, &iacc, &varchild);		
//iacc->accHitTest(m->pt.x, m->pt.y, &varchild); 
BSTR bs; 
iacc->get_accName(varchild, &bs); <-parent node.

Am I missing something? i think i will use this result for next query.

D.D.K-2637 966 Reputation points

2023-05-12T12:10:43.4133333+00:00

Thank you. your codes is exactly what i was looking for.

2 answers

Your answer

D.D.K-2637 966 Reputation points

2023-05-04T04:52:09.6+00:00

Hi, I've seen the following instructions from doc:

to find elements in its own user interface, you must make all UI Automation calls on a separate thread.

but, I have called it like this without success..

Did I use the wrong parameter?

//xyz void test (){ //... thread t([]() { POINT pt{200,300}; CComPtr<IUIAutomation> app; auto hr = CoCreateInstance(__uuidof(CUIAutomation), NULL, CLSCTX_INPROC_SERVER, __uuidof(IUIAutomation), (void**)&app); CComPtr<IUIAutomationElement> element; if (SUCCEEDED(app->ElementFromPoint(pt,&element))) { ... element->get_CurrentName(&name); outputformatW(L"\nname: {}", (wchar_t*)name.m_str); } }); //... }
RLWA32 49,536 Reputation points

2023-05-04T08:20:57.9166667+00:00

I'm not familiar with camtasia.

What does inspect.exe show as the active element in the automation tree when the mouse cursor is positioned where the code would call ElementFromPoint?

Have you tried using the WindowFromPoint function to obtain an HWND for use with ElementFromHandle?
Castorix31 90,681 Reputation points

2023-05-04T08:56:15.83+00:00

If there is a TopMost/Layered window, it is normal.

I use ScreenToGIF and Inspect returns its window too
D.D.K-2637 966 Reputation points

2023-05-04T09:51:47.23+00:00

Hi. it is the transparent camtasia window that obscures everything, this is really annoying.

I hope to have more functionality like :

CComPtr<IUIAutomationElementArray> elements; automation->ElementArrayFromPoint(pt,&elements);

I tried WindowFromPoint(), ChildWindowFromPoint(), RealChildWindowFromPoint() but it doesn't return detail element.
RLWA32 49,536 Reputation points

2023-05-04T11:53:12.53+00:00

You may have success using MSAA to obtain a window handle that you can use with ElementFromHandle.

I used the ScreenToGif application mentioned by @Castorix31 over an instance of Notepad on Win10 22H2. As was noted earlier inspect.exe returned the ScreenToGif window in the automation tree. However, when I switched inspect.exe to MSAA mode (Accessibility) it found Notepad's windows from the mouse position.

I don't know if this will be the case with camtasia but its worth a try.

Update: If you can obtain an IAccessible interface pointer you can call IUIAutomation::ElementFromIAccessible method which doesn't need an HWND.
RLWA32 49,536 Reputation points

2023-05-10T19:51:58.68+00:00

@D.D.K-2637 Any updates to this issue?
D.D.K-2637 966 Reputation points

2023-05-10T22:34:34.24+00:00

Hi @RLWA32 ,

I missed your previous comment.

I just tried MSSA mode now with inspect, it returned the element node level2 of the parent window but the element looking for is on level 5. The problem is that the received attributes of this element seem to be different.

so maybe if i can use this element and switch the state back to automation mode then look for child elements i think it is a solution for now.
D.D.K-2637 966 Reputation points

2023-05-11T04:33:31.4266667+00:00

Hi @RLWA32 ,

I tried the ElementFromIAccessible Method, it seem required for an id to the element. but I haven't been able to determine the focus element (under cursor) so I probably won't know this id.
RLWA32 49,536 Reputation points

2023-05-11T07:47:07.3+00:00

As a starting point try passing CHILDID_SELF to the method. What happens then?

Update:

If you have an IAccessible interface pointer you can also try calling QueryInterface for IOleWindow. If it succeeds you can obtain an HWND to be used with ElementFromHandle. A useful utilty function to simplify this is IUnknown_GetWindow function
D.D.K-2637 966 Reputation points

2023-05-11T09:39:21.88+00:00

Thanks you.

I've seen IAccessible appear on inspect but it's pretty new to me now.

I will try it tonight.
RLWA32 49,536 Reputation points

2023-05-11T10:10:46.6733333+00:00

Your starting point would be to call the AccessibleObjectFromPoint function.
D.D.K-2637 966 Reputation points

2023-05-12T05:35:19.8333333+00:00

Hi @RLWA32 ,

it worked but the result point to the parent element instead focus element .

The code I use:

CComPtr<IAccessible> iacc; VARIANT varchild; varchild.vt = VT_I4; varchild.intVal = CHILDID_SELF; AccessibleObjectFromPoint(m->pt, &iacc, &varchild); //iacc->accHitTest(m->pt.x, m->pt.y, &varchild); BSTR bs; iacc->get_accName(varchild, &bs); <-parent node.

Am I missing something? i think i will use this result for next query.
D.D.K-2637 966 Reputation points

2023-05-12T12:10:43.4133333+00:00

Thank you. your codes is exactly what i was looking for.

Answer 1

RLWA32 49,536

Assuming you are using ATL. Following snippet assumes pt contains the screen coordinates of the mouse cursor and pAuto contains an IUIAutomation interace pointer.

CComPtr<IUIAutomationElement> pElement;
CComPtr<IAccessible> pAcc;
CComVariant vChild;
HRESULT hr;

hr = AccessibleObjectFromPoint(pt, &pAcc, &vChild);
if (SUCCEEDED(hr))
{
    ATLASSERT(V_VT(&vChild) == VT_I4);
    hr = pAuto->ElementFromIAccessible(pAcc, V_I4(&vChild), &pElement);
    if (SUCCEEDED(hr) && pElement)
    {
        // Insert code
    }
}

RLWA32 49,536 Reputation points

2023-05-12T14:25:05.0866667+00:00

@D.D.K-2637

Thank you. your codes is exactly what i was looking for.

Based on the above I converted my comment to an Answer that can be accepted.

Answer 2

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Share via

How to get ElementFromPoint?

2 answers

Your answer