Finding window element using MSAA

Laila 0 Reputation points
2024-06-14T20:07:42.4333333+00:00

I'm trying to detect the "Previous" and "Next" buttons of the Spofity MiniPlayer window using MSAA, however, sometimes these buttons don't get detected, I don't understand why, it's like luck, I need to reopen the window uncountable times and it randomly gets detected.

It's not a timing or focus issue, I can wait for hours, resize the window, click on every button, and it randomly decides to detect the control, once it's detected it always succeeds as long I don't reopen the window, if I do, then its "luck" again.

Using the Microsoft tool inspect, I caught a case where the controls weren't getting detected.

With the option MSAA:

inspect_vURdPNfIS3

When I change the option to UI Automation the buttons then get detected:inspect_JfIBgse2E4

It wasn't necessary to focus or modify anything on the window, just changing the option it got detected.

If I change back to MSAA it now detects the controls correctly.

However, I would like to use MSAA as UI Automation doesn't work properly when the window is on the background or hidden.

This is how I'm trying to detect the button:

#include <iostream>
#include <Windows.h>
#include <oleacc.h>
#pragma comment(lib, "oleacc.lib")

void printElementInfo(IAccessible* pAcc, long childId)
{
    VARIANT varChild;
    varChild.vt = VT_I4;
    varChild.lVal = childId;
    BSTR bstrName;
    if (pAcc->get_accName(varChild, &bstrName) == S_OK)
    {
        std::wstring ws(bstrName);
        if (ws == L"Previous") // Button found!!
            OutputDebugStringW(ws.c_str());
        SysFreeString(bstrName);
    }
}

void iterateOverElements(IAccessible* pAcc)
{
    long childCount, returnCount;
    pAcc->get_accChildCount(&childCount);
    if (childCount > 0)
    {
        VARIANT* pArray = new VARIANT[childCount];
        if (AccessibleChildren(pAcc, 0L, childCount, pArray, &returnCount) == S_OK)
        {
            for (int i = 0; i < returnCount; i++)
            {
                if (pArray[i].vt == VT_DISPATCH)
                {
                    IDispatch* pDisp = pArray[i].pdispVal;
                    IAccessible* pChild = NULL;
                    if (pDisp->QueryInterface(IID_IAccessible, (void**)&pChild) == S_OK)
                    {
                        printElementInfo(pChild, CHILDID_SELF);
                        iterateOverElements(pChild);
                        pChild->Release();
                    }
                    else
                        OutputDebugStringW(L"Failed to query interface for child.\n");
                }
                else if (pArray[i].vt == VT_I4)
                    printElementInfo(pAcc, pArray[i].lVal);
                else
                    OutputDebugStringW(L"Unknown child type.\n");
            }
        }
        else
            OutputDebugStringW(L"Failed to get accessible children.\n");
        delete[] pArray;
    }
}

int main()
{
    CoInitialize(NULL);
    HWND hwnd = FindWindow(L"Chrome_WidgetWin_1", NULL);
    if (!hwnd)
    {
        OutputDebugStringW(L"Failed to find window.\n");
        return -1;
    }
    IAccessible* pAcc = NULL;
    if (AccessibleObjectFromWindow(hwnd, OBJID_WINDOW, IID_IAccessible, (void**)&pAcc) != S_OK)
    {
        OutputDebugStringW(L"Failed to get accessible object from window.\n");
        return -1;
    }
    iterateOverElements(pAcc);
    pAcc->Release();
    CoUninitialize();
    return 0;
}

This issue can be reproduced on any browser: login on the web Spotify, on the bottom-right next to the sound button, there's a button "Open MiniPlayer" it will open a new window called "Spotify MiniPlayer".

You can interact with the window like resizing, clicking on its buttons, and then run the code, it sometimes detects the button, sometimes doesn't, and when don't the only solution is to reopen the window until it works.

In the cases that the code doesn't detect the button, if I inspect the window with the UIA option it then starts detecting the button. I also noted that after inspecting the window whenever I reopen it, MSAA always works in detecting the buttons, even the window being deleted/recreated each time.

Does UIA call something that makes the controls be initialized(?) or something like that?

Windows API - Win32
Windows API - Win32
A core set of Windows application programming interfaces (APIs) for desktop and server applications. Previously known as Win32 API.
2,517 questions
C++
C++
A high-level, general-purpose programming language, created as an extension of the C programming language, that has object-oriented, generic, and functional features in addition to facilities for low-level memory manipulation.
3,632 questions
{count} votes