Custom UI Automation Providers in Depth: Part 5
In part 4, we added the Value Pattern to our custom provider. Now we’re going to learn how to represent items that are contained within our provider. The sample code for this section is here.. In our example, we want to represent each bar of the tri-color picker as a separate UI element. In UIA, these contained items are called fragments.
The word fragment sounds a bit exotic, but you’ve certainly seen fragments before. In this image of my Computer folder, you can see at least two: the folder pane on the left has a fragment for each folder, and the items pane on the right has a fragment for each of my hard disk volumes. From an Accessibility perspective, the customer certainly wants to know the details of each of these items – it’s no good to say, “There’s a list of folders here” and leave it at that!
Considering the example of the folder tree, you can see that the root of the tree is special. The TreeView control, in this case, knows about all of its fragments, not just its immediate children. In UIA, the special fragments that knows about all of its descendants is called a fragment root. To provide an example: if I have a TreeView that is showing two levels of folders (that it, both C:\Child and C:\Child\Grandchild), the parent of the Grandchild fragment is just the ordinary fragment for Child. But the parent of the Child fragment is special: it corresponds to the whole TreeView control and is the fragment root.
So far, our custom provider has been representing a full HWND. The HWND provider within UIA has been providing a lot of information on our behalf, based on the HWND itself. Once we decide to represent something smaller than a full HWND, we have to answer a number of new questions that we did not have to handle before:
- Where is the fragment?
- How does it relate to other fragments within the Accessibility tree? That is, who are its siblings? Its children?
- Which fragment has keyboard focus?
- If the caller provides a point, which fragment contains that point? (hit-testing)
- How would UIA uniquely identify the fragment?
There’s really no way around these questions. Once you get into fragments of an HWND, UIA has to ask you to describe them. So, let’s see how we answer these questions.
As in previous posts, I’m going to use base classes to make a distinction between functionality that most providers would need and functionality unique to a specific provider. This ends up saving a reasonable amount of work, as you’ll see. I created two classes to represent my fragment, then: BaseFragmentProvider and TriColorFragmentProvider. The BaseFragmentProvider will capture the parent for each fragment and the fragment root for each fragment. (In our case – a one-level tree – these are the same, but in a deeper tree they wouldn’t be.) The TriColorFragmentProvider will also hold the value for which this is the fragment (Red, Yellow or Green).
All fragments are also simple providers. So, all of the work we described previously for simple providers needs to be done for fragments. I derived BaseFragmentProvider from BaseSimpleProvider to make this explicit. And then I had TriColorFragmentProvider use the static property bag trick that I outlined in part 3:
public TriColorFragmentProvider(TriColorControl control, IRawElementProviderFragmentRoot root, TriColorValue value)
: base(root /* parent */, root /* fragmentRoot */)
{
this.control = control;
this.value = value;
// Populate static properties
//
// In a production app, Name should be localized
AddStaticProperty(AutomationElementIdentifiers.NameProperty.Id, this.value.ToString());
AddStaticProperty(AutomationElementIdentifiers.ControlTypeProperty.Id, ControlType.Custom.Id);
// In a production app, LocalizedControlType should be localized
AddStaticProperty(AutomationElementIdentifiers.LocalizedControlTypeProperty.Id, "tri-color item");
AddStaticProperty(ProviderDescriptionId, "UIASamples: Tri-Color Fragment Provider");
AddStaticProperty(AutomationElementIdentifiers.AutomationIdProperty.Id, this.value.ToString());
AddStaticProperty(AutomationElementIdentifiers.IsKeyboardFocusableProperty.Id, false);
AddStaticProperty(AutomationElementIdentifiers.IsControlElementProperty.Id, true);
AddStaticProperty(AutomationElementIdentifiers.IsContentElementProperty.Id, false);
}
Returning to our fragment questions: fragments must implement IRawElementProviderFragment, which specifies the list of questions we must answer as a set of methods. We’ll first provide a runtime ID, which is how UIA identifies elements. A runtime ID is an array of integers. Runtime IDs can change from run to run, but they do need to be unique during the lifetime of the object. UIA considers two elements to be identical if and only if their runtime IDs are identical. For a full HWND, the runtime ID is generated for you automatically based on the HWND’s numeric value. To identify our fragment, we can use any unique integer we wish. I chose the fragment’s numeric index (1-3, for the tricolor control). To make it unique, we want to concatenate our index to the HWND’s numeric value. UIA makes this easy: if you start your runtime ID with a special AppendRuntimeId value, UIA will do the append for you:
public override int[] GetRuntimeId()
{
int[] runtimeId = new int[2];
runtimeId[0] = AutomationInteropProvider.AppendRuntimeId;
runtimeId[1] = (int)this.value;
return runtimeId;
}
Next up is the bounding rectangle, which should be specified in screen coordinates. My TriColorControl exposes a method to get the client rectangle for a specific value, so I’ll just call that and convert to screen coordinates:
// Get the bounding rect by consulting the control.
public override System.Windows.Rect BoundingRectangle
{
get
{
// Bounding rects must be in screen coordinates
System.Drawing.Rectangle screenRect = this.control.RectangleToScreen(
this.control.RectFromValue(this.value));
return new System.Windows.Rect(
screenRect.Left, screenRect.Top, screenRect.Width, screenRect.Height);
}
}
The next method to implement is GetEmbeddedFragmentRoots(). This allows a fragment to declare that it actually contains a fragment root of its own. We don’t have this situation, so we’ll just say no by returning null. (A full discussion of this topic would be good matter for a full post.)
Next comes an easy method: get_FragmentRoot(). This just returns the fragment root for a given fragment. Since all fragments do this identically, I made the fragment root a required parameter for the BaseFragmentProvider and put the implementation of this method in that class:
// Return the fragment root: the fragment that is tied to the window handle itself.
// Don't override, since the constructor requires the fragment root already.
public IRawElementProviderFragmentRoot FragmentRoot
{
get { return this.fragmentRoot; }
}
We have one more question to answer: how does this fragment relate to other fragments? That is the role of the Navigate() method, which allows UIA to ask for a fragment’s parent, siblings, and children. When I tried to implement this, I realized that this is one method with several distinct questions: “Who is your parent?” is distinct from “Who is your first child?” Using the principle of “every function should do one thing,” I implemented Navigate() in BaseFragmentProvider as a simple routing function:
// Routing function for going to neighboring elements. We implemented
// this to delegate to other virtual functions, so don't override it.
public IRawElementProviderFragment Navigate(NavigateDirection direction)
{
switch (direction)
{
case NavigateDirection.Parent: return this.parent;
case NavigateDirection.FirstChild: return GetFirstChild();
case NavigateDirection.LastChild: return GetLastChild();
case NavigateDirection.NextSibling: return GetNextSibling();
case NavigateDirection.PreviousSibling: return GetPreviousSibling();
}
return null;
}
The base class can handle the Parent trivially, but it isn’t sure about the others, so it delegates to virtual functions that return null. Our color fragments really don’t have children, so returning null for FirstChild and LastChild is correct, but they do need to report their next and previous siblings. At this point, I just have my TriColorFragmentProvider override the base class methods I just added to answer these questions:
// Return the fragment for the next value
protected override System.Windows.Automation.Provider.IRawElementProviderFragment GetNextSibling()
{
if (!TriColorValueHelper.IsLast(this.value))
{
return new TriColorFragmentProvider(
this.control,
this.fragmentRoot,
TriColorValueHelper.NextValue(this.value));
}
return null;
}
// Return the fragment for the previous value
protected override System.Windows.Automation.Provider.IRawElementProviderFragment GetPreviousSibling()
{
if (!TriColorValueHelper.IsFirst(this.value))
{
return new TriColorFragmentProvider(
this.control,
this.fragmentRoot,
TriColorValueHelper.PreviousValue(this.value));
}
return null;
}
There’s one last method to implement on IRawElementProviderFragment: SetFocus(). UIA calls this to set keyboard focus to a fragment. Our tri-color fragments cannot take keyboard focus, so that method just does nothing. And with that, we’re done with fragments.
One last task remains: we need to make the TriColorProvider itself into a fragment root by having it implement IRawElementProviderFragmentRoot. All fragment roots are also fragments, so it has to implement IRawElementProviderFragment, too. I will use my base class strategy one more time and create a base class for fragment roots: BaseFragmentRootProvider. The base class will take care of a couple of chores: it will implement GetRuntimeId() and get_BoundingRectangle() by returning nothing, confident that the HWND provider will do the work for us. I went on to re-derive TriColorProvider from the BaseFragmentRootProvider:
/// <summary>
/// Provider for the TriColor control itself.
/// </summary>
public class TriColorProvider : BaseFragmentRootProvider, IValueProvider
{
As a fragment, TriColorProvider must override any of the appropriate Navigate methods, and since it does have children, it will override the GetFirstChild() and GetLastChild() methods:
protected override IRawElementProviderFragment GetFirstChild()
{
// Return our first child, which is the fragment for Red
return new TriColorFragmentProvider(this.control, this, TriColorValue.Red);
}
protected override IRawElementProviderFragment GetLastChild()
{
// Return our last child, which is the fragment for Green
return new TriColorFragmentProvider(this.control, this, TriColorValue.Green);
}
Now we can get on to the real work of being a fragment root.
Actually, fragment roots only have two methods to implement beyond what any other fragment does. First, they have to answer the question, “Which fragment has the keyboard focus?” In our case, no fragment does, so the GetFocus() method can just return null. (UIA is smart enough to realize that if the HWND has focus and no fragment does, the fragment root itself must have focus.)
The second question is, “Which fragment contains a given point?,” also known as hit-testing. The method to do this is called ElementProviderFromPoint(). The fragment root is expected to hit-test all the way down to the smallest fragment containing the point, so even if the tree contains multiple layers, this method needs to drill all the way down. The algorithm here ends up being:
- Decide which of your sub-elements was hit, if any.
- Create a fragment provider and return it if there was a hit.
- Return yourself (or null) if there wasn’t a hit.
Here’s how it looks in TriColorProvider:
// Check to see if the passed point is a hit on one of our children
public override IRawElementProviderFragment ElementProviderFromPoint(double x, double y)
{
// Convert screen point to client point
System.Drawing.Point clientPoint = this.control.PointToClient(
new System.Drawing.Point((int)x, (int)y));
// Have the control do a hit test and see what value this is
TriColorValue value;
if (this.control.ValueFromPoint(clientPoint, out value))
{
// Return the appropriate fragment
return new TriColorFragmentProvider(this.control, this, value);
}
return null;
}
And that’s it – I’ve finished making TriColorProvider into a fragment root.
When I build and run my sample and Inspect it, I notice several differences:
- The TriColorProvider has children now, three of them.
- If I hover over a color bar, Inspect will highlight the fragment, not the whole control.
- The fragment has a whole bunch of its own properties, reflecting what I built into TriColorFragmentProvider.
Next time: Now that I have a fragment structure, I can implement the Selection pattern on my control.
Comments
Anonymous
July 14, 2010
Great post. Can you also provide an example of multiple fragment roots in a single custom control?Anonymous
July 15, 2010
Thanks, Kevin. Did you have a specific scenario in mind for this? Multiple fragent roots usually come up when you have one framework hosting another framework; I'm struggling to think of a good example that would make a straightforward sample.Anonymous
July 27, 2010
Hi. In your code, I found that you simply return a new fragment provider in Navigate(). But in the SDK samples, a container of fragment providers in the control is maintained and referenced depending on the request(previous,next,first,last). Is there any reason for your case in not doing so?Anonymous
July 28, 2010
Hi, Dayn, Either way is correct -- it's really a decision about caching strategy. When you have lots of incoming requests, keeping a container of fragments is more efficient, since you avoid some heap allocations. The trouble is, when the requests stop, you end up holding data in memory unnecessarily. My sample is optimized for relatively few requests -- I only allocate on demand, but if there are lots of demands, I'll do lots of allocations. It ends up being a performance tuning decision.Anonymous
July 28, 2010
Thanks for your clarification, Bernstein. Right now, I've been trying to add fragment providers to items belonging to a MFC CCheckListBox control. My goal is simply to add a toggle pattern for each item to reflect its checked status. But I always end up with a duplicate set of items in UISpy, meaning the list items and the new fragment providers are all displayed together. I wonder if you have any explanation for this behavior?Anonymous
July 29, 2010
Hi, Dayn, Can I suggest that you bring this question to the MSDN Forum for Accessibility? Comments are a hard place to do Q&A. social.msdn.microsoft.com/.../threads Thanks, MichaelAnonymous
February 20, 2012
Im trying hands on writing providers for MFC based SDI application . In that im writing providers for menus and manu buttons. Im using VS 2010 and those menus cannot be detected in UI spy. Plse help.Anonymous
February 21, 2012
Hi, Venkatesh, Please see the previous comment about the MSDN forum -- comments are a hard place to do Q&A. Thanks, MichaelAnonymous
April 19, 2015
Hi Michael, the links to the samle code doesnt work anymore, can you supply us with new links?