Finding window element using MSAA
I'm trying to detect the "Previous" and "Next" buttons of the Spofity MiniPlayer window using MSAA, however, sometimes these buttons don't get detected, I don't understand why, it's like luck, I need to reopen the window uncountable times and it randomly gets detected.
It's not a timing or focus issue, I can wait for hours, resize the window, click on every button, and it randomly decides to detect the control, once it's detected it always succeeds as long I don't reopen the window, if I do, then its "luck" again.
Using the Microsoft tool inspect, I caught a case where the controls weren't getting detected.
With the option MSAA:
When I change the option to UI Automation the buttons then get detected:
It wasn't necessary to focus or modify anything on the window, just changing the option it got detected.
If I change back to MSAA it now detects the controls correctly.
However, I would like to use MSAA as UI Automation doesn't work properly when the window is on the background or hidden.
This is how I'm trying to detect the button:
#include <iostream>
#include <Windows.h>
#include <oleacc.h>
#pragma comment(lib, "oleacc.lib")
void printElementInfo(IAccessible* pAcc, long childId)
{
VARIANT varChild;
varChild.vt = VT_I4;
varChild.lVal = childId;
BSTR bstrName;
if (pAcc->get_accName(varChild, &bstrName) == S_OK)
{
std::wstring ws(bstrName);
if (ws == L"Previous") // Button found!!
OutputDebugStringW(ws.c_str());
SysFreeString(bstrName);
}
}
void iterateOverElements(IAccessible* pAcc)
{
long childCount, returnCount;
pAcc->get_accChildCount(&childCount);
if (childCount > 0)
{
VARIANT* pArray = new VARIANT[childCount];
if (AccessibleChildren(pAcc, 0L, childCount, pArray, &returnCount) == S_OK)
{
for (int i = 0; i < returnCount; i++)
{
if (pArray[i].vt == VT_DISPATCH)
{
IDispatch* pDisp = pArray[i].pdispVal;
IAccessible* pChild = NULL;
if (pDisp->QueryInterface(IID_IAccessible, (void**)&pChild) == S_OK)
{
printElementInfo(pChild, CHILDID_SELF);
iterateOverElements(pChild);
pChild->Release();
}
else
OutputDebugStringW(L"Failed to query interface for child.\n");
}
else if (pArray[i].vt == VT_I4)
printElementInfo(pAcc, pArray[i].lVal);
else
OutputDebugStringW(L"Unknown child type.\n");
}
}
else
OutputDebugStringW(L"Failed to get accessible children.\n");
delete[] pArray;
}
}
int main()
{
CoInitialize(NULL);
HWND hwnd = FindWindow(L"Chrome_WidgetWin_1", NULL);
if (!hwnd)
{
OutputDebugStringW(L"Failed to find window.\n");
return -1;
}
IAccessible* pAcc = NULL;
if (AccessibleObjectFromWindow(hwnd, OBJID_WINDOW, IID_IAccessible, (void**)&pAcc) != S_OK)
{
OutputDebugStringW(L"Failed to get accessible object from window.\n");
return -1;
}
iterateOverElements(pAcc);
pAcc->Release();
CoUninitialize();
return 0;
}
This issue can be reproduced on any browser: login on the web Spotify, on the bottom-right next to the sound button, there's a button "Open MiniPlayer" it will open a new window called "Spotify MiniPlayer".
You can interact with the window like resizing, clicking on its buttons, and then run the code, it sometimes detects the button, sometimes doesn't, and when don't the only solution is to reopen the window until it works.
In the cases that the code doesn't detect the button, if I inspect the window with the UIA option it then starts detecting the button. I also noted that after inspecting the window whenever I reopen it, MSAA always works in detecting the buttons, even the window being deleted/recreated each time.
Does UIA call something that makes the controls be initialized(?) or something like that?