Introduction

Around this time last year, I began an experiment into how a Microsoft Store app might leverage many accessibility-related features of Windows. By creating a demo solitaire game called Sa11ytaire, I, with the help of my colleague Tim, showed how straightforward it can be to build an app which can be used with many input and output mechanisms. Our goal was to encourage all app builders to consider how their own apps can be efficiently used by as many people as possible. I published my first articles describing the technical steps for building the app at Microsoft Windows UI Automation Blog.

Since then I contained experimenting, with the goal of demonstrating how Azure Cognitive Services might help to enable exciting new scenarios in apps. I really found it fascinating to explore how such things as speech to text, image recognition, language understanding and bots might help a game become more useable to more people. I continued documenting the technical aspects of updating the demo game to incorporate use of Azure Cognitive Services, at my own LinkedIn page, Barker's articles.

Below I've included links to the full set of twelve Sa11ytaire articles. Or rather, I've included links to what's currently the full set. The Sa11ytaire Experiment never ends.

All the best for 2019 everyone!

Guy

The articles

1. The Sa11ytaire Experiment: Part 1 – Setting the Scene

Figure 1: The Windows Eye Control UI showing over the Sa11ytaire app, and being used to move a 10 of Clubs over to a Jack of Hearts.

2. The Sa11ytaire Experiment – Enhancing the UIA representation

Figure 2: The Inspect SDK tool reporting the UIA properties of an upturned card element, with properties relating to Name, ControlType, HelpText, and the Toggle pattern highlighted.

3. Sa11ytaire on Xbox: Let the experiment begin!

Figure 3: The Sallytaire game being played on the Xbox, with a switch device plugged into an Adaptive Controller. A ten of diamonds is being moved onto a jack of clubs.

4. The Sa11ytaire Experiment – Reacting to Feedback Part 1: Keyboard efficiency

Figure 4: The Sa11ytaire app running on a Surface Book with an external number pad connected.

5. The Sa11ytaire Experiment – Reacting to Feedback Part 2: Localization and Visuals

Figure 5: The high contrast Sa11ytaire app being played with a footswitch plugged into a Bluetooth-connected Xbox Adaptive Controller.

6. The Sa11ytaire Experiment - End of Part 1

Figure 6: The Sa11ytaire app running on a Surface Book, surrounded by a variety of input devices.

7. The Sa11ytaire Experiment Part 2: Putting the AI in Sa11ytAIre

Figure 7: The Sa11ytaire app showing the results of the Azure Language Understanding service. An utterance of "Grab the 4 of clubs and put it on the 3 of clubs" was matched with an intent of "MoveCard" with a confidence of 0.99965173.

8. Using Azure Custom Vision services to play Sa11ytaire by presenting a playing card to the game

Figure 8: The physical 2 of Diamonds card being held up in front of a Surface Book running the Sa11ytaire app.

9. Delivering a more helpful game experience through Azure Language Understanding, Q&A, and Search

Figure 9: The Sa11ytaire app showing the result of a Bing Web Search in response to the player asking the question "Where can I buy Braille playing cards?" at the app.

10. Adding an Azure Bot to a game to enable a more natural experience

Figure 10: The Sa11ytaire app hosting the Sa11y bot using the Azure Bot Service's web chat UI

11. Interacting with an Azure bot directly from a Windows Store app

Figure 11: The Sa11ytaire app interacting directly with the Sa11y bot, and using Speech to Text and Text to Speech to interact with the player.

12. The Sa11ytaire Experiment - End of Part 2

Figure 12: The Sa11ytaire app running on a Surface book, showing Sa11y the bot. By the device are playing cards and a Dalek wearing headphones.

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Introduction

Earlier this year, at The Sa11ytaire Experiment: Part 1 – Setting the Scene I described how a colleague and I built a Microsoft Store app which explored a variety of input and output mechanisms that could be used to play a solitaire game. I felt this was a fascinating topic, involving input mechanisms such as speech, eye control, switch device, keyboard access keys, and also considering the most effective Narrator experience we could build. While the app was far from being complete, (for example, it didn't provide a usable experience on small screens or when different themes were active,) it had reached a point where we could start tuning the app based on feedback.

Since we built the app, Microsoft announced the Xbox Adaptive Controller, and so it was inevitable that I wasn't going to be able to relax until I'd played Sa11ytaire with an Adaptive Controller. This post describes the steps I took to get Sa11ytaire running on an Xbox. I've never built an Xbox game before, and so I mention a few places where there were hiccups along the way, (mostly caused by me not paying much attention to all the helpful resources available at docs.microsoft.com).

Please do send feedback on the Sa11ytaire Xbox experience. I know there are many things that could be improved with the app, (for example the visuals are still rudimentary,) but I'm hoping the current switch device and Narrator experiences provide the foundation for a usable experience.

The Sa11taire Xbox game is available at Sa11ytaire at the Microsoft Store.

Figure 1: The Sallytaire game being played on the Xbox, with a switch device plugged into an Adaptive Controller. A ten of diamonds is being moved onto a jack of clubs.

A few other thoughts

Before we get started, a few additional comments.

Is the game really usable?

There are a fair number of known bugs and limitations in the game, but I do believe it's worth sharing it out as it is, in order to help us focus on the top priority changes first. The most significant constraint with the switch device experience, is that on Xbox, the switch scan speed cannot be adjusted through switch input. So the speed needs to be set using something other than the switch device.

What about Magnifier on the Xbox?

As far as I know, Magnifier works with Sa11ytaire just as it does with any other app. I couldn't find a way to have Magnifier follow keyboard focus in the app as the Magnifier feature can on the desktop, so the view needs to be manually updated to bring the cards of interest into view.

What about colors used in the app?

Color usage is a critical part of building an app. Providing sufficient contrast, and respecting customers' choice of light-on-dark or dark-on-light, is such a fundamental concept. That said, I didn't focus on that for this porting of the game to Xbox, simply due to time constraints. The visuals today provide a means to learn about the switch device and Narrator experiences, and we can prioritize updates to the app based on your feedback.

What about my Windows Phone?

Since we'd built a UWP XAML app, it should work on any Windows 10 device right? I was porting the Windows desktop app to the Xbox, and who knows, maybe the Sa11ytaire experiment will move to Hololens at some point. But what about my Windows Phone? When I'm sitting on the 255 bus, why wouldn't I want to play the game and consider the next steps for improving it? Well, it looks like I'm out of luck. I tried deploying the app to my Microsoft Lumia 950, and Visual Studio told me the phone's OS was too old. Perhaps I could address that my changing the Sa11ytaire app to build for the older OS, but there's no way I'm doing that. One of the key points of Sa11ytaire app is to explore how to get the most out of the great new accessibility-related features available in whatever the latest version of Windows 10 is. And as far as I can tell, there is no later version of the OS available for my phone. So I think it might be time for me and my Windows Phone to part ways. That's too bad really, I'll miss it.

That last point raises the question: Should the app be turned into a Xamarin app, enabling it to run on multiple mobile platforms? That is a tempting idea, but I'm not sure if a Xamarin app running on Windows 10 can leverage all the latest accessibility features of Windows. The Sa11ytaire app makes some very deliberate use of the UIA Notification event through the UWP XAML platform, as part of trying to deliver the most efficient Narrator experience possible. If I can't do that with a Xamarin app today, then I'll continue with the regular UWP XAML app.

Porting the desktop app to the Xbox

Typically, I was going to do the least amount of preparation that I could while porting the game to Xbox. I tell myself that that's because I just don't have time to read all the material out there which explains what I should be doing. But I have been called a slacker in the past, so perhaps that's the real problem. Either way, I wondered if I could simply publish the existing desktop game as-is for the Xbox, and maybe it would "just work".

So the first step was publishing for Xbox. At the Services section for the Sa11ytaire app at the Dev Center, I selected "Xbox Live" and viewed all the things I could do there. The only thing I actually did there was invoke the Settings button, and then say that the app supports both Desktop and Xbox One. Back in the main Properties for the app, the app was already declared to be a "Card + board" game, and since Xbox seems to require that the app is a game, that was all fine. I set the "Game settings" to be the most basic they could be. That is, single player for PC and Xbox. (I left the Broadcasting setting checked as that seemed like that might be handy if in the future someone wants to provide feedback through a broadcast.)

Having done that, an error string appeared on the submission page, saying "Document: accesspolicies.xml Error: AccessPoliciesRequired The access policies document is not present in the config set. This document is required for all publish operations". I searched high and low for information about that, and in the end got help from someone in Xbox support. It turned out that all I had to do was press the "Test" button on the Xbox Live Services page at the Dev Center, and the error went away. I suspect others might not hit this error as I did, as I think my dev account might have been in an unusual state. (The account was created way back, perhaps when the process for setting up Xbox accounts was still evolving.) But if you ever do hit that mysterious error, try invoking the "Test" button. (You would have done that already of course, if you were actually testing your app on an Xbox before publishing it…)

At that point I published the app, and to my delight soon after found I could download the app on my Xbox. It technically worked, but there were a couple of things I'd need to do to make it usable. The first related to preventing some UI clipping, and so needed to support a smaller screen resolution that the app currently supported. Addressing that was routine UWP XAML UI work, including making sure I reduced minimum widths for some elements. And while the results still led to some text clipping, it would do for now. The second point related to input via the Xbox controller. By default the controller was moving the pointer on the screen, and instead I needed it to move focus between the cards in the app. In order to address that, I added the following to the app, after which, the controller's D-pad moved me left/right/up/down just great.

RequiresPointerMode = ApplicationRequiresPointerMode.WhenRequested;

(For more details, visit ApplicationRequiresPointerMode.)

While making the above changes, I did re-publish the app a couple of times. But then after one attempt, I was told the app submission was rejected due to:

"Create at least one active user and allow the user to sign the user in to Xbox. Display the user's Xbox gamertag as the primary display and profile name. Please add this behavior and resubmit your game."

It was becoming clear I couldn't keep publishing the app and hoping for the best. I needed to develop and test using my own Xbox, and publish when I was reasonably confident that the app would actually work. So I took a look at a variety of resources, including:

• Xbox One Developer Mode activation
• Device Portal for Xbox
• Xbox Live sandboxes introduction
• Get started developing an Xbox Live Creators Program title with Visual Studio
• Authorize Xbox Live Accounts for Testing in your environment

Setting up my Xbox in dev mode and connecting to it through the Dev Portal from a browser on my laptop was surprisingly straightforward once I'd got used to configuring the Xbox's Remote Access Settings, as described at Introduction to Xbox One tools. (I found that page after first encountering the "This site is not secure" thing described there, and I have to say, I did find that a little disconcerting at the time.)

Having effectively turned my Xbox into a dev machine, the next step was to debug Sa11ytaire running on it. I learned about deploying the app from Visual Studio to the Xbox, as described at Set up your UWP on Xbox development environment, and successfully entered the VS pin when requested. Following that, I got stuck. All my attempts to actually deploy the app ended with me being told "Failed to launch remote debugger with the following error: 'Command failed: 0x800705b4'". I then spent 90 minutes scouring the web for tips on how to deal with this, but was not successful. (I must admit that by the end of that, I felt my original attempts to publish the app without testing it, weren't perhaps that outrageous after all.) Since that day when I hit the 0x800705b4 problem, it was suggested to me that given that this is a ERROR_TIMEOUT, I should consider moving to a wired, rather than wireless connection. And in fact after reviewing Create your first app again, it does say "A wired network connection is recommended". I've yet to try that, but it's next on my list of things to do…

With my attempt to debug the app on hold, I instead installed the app from my laptop through the Dev Portal, and ran it on the Xbox. Not being able to debug made diagnosing problems rather inefficient, but I could still make a ton of progress anyway. The first interesting thing I learned was that the app failed to start. This was because I'd not set up the sandboxing as required. But once I'd set up the Xbox's sandbox to match that of the Sa11ytaire app I was installing, the app started fine.

The next step was to add the Xbox Live sign-in code for app start-up, and so copied in the C# code sample at Authentication for UWP projects. Initially, this seemed to work great. I ran the app, worked through various sign-in-related UI on app start-up, and displayed the player tag in the UI. At this point, I thought I was good to go. I then found that after playing the game for a while, it would crash at apparently random moments. As I recall, I was encountering a NullReferenceException beneath Microsoft.Xbox.Services.dll, with none of my own code shown in the callstack. I have no clue what was going on there, but after some experimenting, I found that if I removed the creation of the unused XboxLiveContext object, the app stopped crashing. As such, for now, I'll not be creating an XboxLiveContext object.

And so I'd reached the point where I had an Xbox app that could be published to the Microsoft Store, and could charge ahead on considering the switch control and Narrator experiences.

Switch device control

The switch device control of the Sa11ytaire app on the desktop, worked through Space key input. I tested this with a switch device and an adaptor, such that in response to a switch device press, the app would receive the Space key input. (Limiting the type of input to the Space key on the desktop was only due to time constraints.) In order to test this out on the Xbox, I hooked up the Y button to toggle the state of switch control, and pressed the Space key on my controller's chatpad. As a result, the switch control scan highlight cycled through the UI, and I could play the game with the controller.

But supporting only a chatpad's Space key input is not sufficient for the app on Xbox, so I hooked up the left and right bumpers to control the app when switch control is active. I did this by adding a Window.Current.CoreWindow.KeyDown handler, and responding to presses of VirtualKey.GamepadLeftShoulder/GamepadRightShoulder. (The original Space key action was triggered in response to the page's overridden OnPreviewKeyDown() being called.) All in all, that seemed pretty straightforward.

One additional modification that was required related to interaction with standard buttons in a ContentDialog. (For example, the "Are you sure you want to restart the game?" dialog.) When switch device control is enabled, when one of those dialogs appears, the app simply moves keyboard focus between the buttons, and whenever the Space key is pressed, then the focused button gets invoked. So the fact that this worked on the desktop was a side-effect of having the app react to a Space key press when switch control is active. In order to make this work on Xbox, I added the following code, to be run when a bumper is pressed and a ContentDialog is up. (Sorry about the dodgy indentation of the code. The blog site won't seem to let me fix that.)

var buttonWithFocus = FocusManager.GetFocusedElement() as Button;

if (buttonWithFocus != null)

{

ButtonAutomationPeer peer = FrameworkElementAutomationPeer.FromElement(buttonWithFocus) as ButtonAutomationPeer;

if (peer != null)

{

peer.Invoke();

}

}

I don't actually remember building a UWP XAML app that invokes one of its own buttons through UIA, but it seemed to work ok. (Note that I've not yet had a chance to update the app such that its appbar UI is usable with an Xbox controller.)

Now I could get to the bit that I was really excited about: Playing the game with the Xbox Adaptive Controller. Playing the game using a bumper or any other specific button on the controller might be fine for some players, but I want to make the game playable for as many players as I can. I've pre-ordered an Adaptive Controller, but it won't arrive for a while yet. So a colleague kindly lent me a device, and I tried it out. The Adaptive Controller supports customization such that its big buttons can be configured to effectively become other buttons, and I did consider doing that. But instead, I just paired the Adaptive Controller with my Xbox, plugged a switch device into the back of it, and hey-presto, I could play Sa11ytaire on my Xbox with a switch device. That was just fantastic!

I can't wait for my own Adaptive Controller to arrive now, so I can get familiar with all that it can do.

Figure 2: The Sa11ytaire app being played with the right bumper of a Xbox controller. A ten of diamonds and nine of clubs are being moved on to a jack of spades.

Narrator

As far as Narrator goes, I pretty much knew what the experience was going to be like with Sa11ytaire on the Xbox. After all, the same Narrator, UIA and UWP XAML is running on both the Windows desktop and Xbox. We'd built Sa11ytaire on the desktop to have Narrator make specific announcements which we felt would be valuable to the player, and those announcements would also be made on the Xbox. That said, we did adjust the experience, simply due to further consideration while we played the game.

For example, I added an announcement to confirm that the game had been restarted by the player. When I originally added that I felt that the announcement was too important to be truncated by another announcement, and so when I raised the related notification event, I supplied ImportantAll. That turned out to be the wrong thing to do. Say a player restarts the game many times in quick succession. With my change, an entire "Game restarted" announcement will be made for each restart of the game. The game and the player may well be ready for further play long before the full set of announcements have completed. So that's a tedious experience. As it happens, Tim caught this error and fixed it by replacing ImportantAll with ImportantMostRecent. (Thanks Tim!)

Another change Tim made relates to the announcement on whether a move is available between dealt card piles. Previously if no move is available there would be no announcement, and so the player might wonder if they really issued the command to learn about available moves or not. So now, if no moves are available, Narrator announces "No move is available".

By the way, I also removed some of the UIA Help properties from a few elements. When the game was originally built, I think I went over the top with the Help properties. I was effectively stuffing the instructions on how to play the game into the Help properties, and I don't think that's appropriate. It led to far too much information being announced simply when trying to play the game.

I'd say the most interesting aspect of using Narrator when porting Sa11ytaire to the Xbox, was how the controller should be used as a form of input at the game. With the D-pad working great, and the A button interacting as required with the focused card by default, the game was usable. I could move to one card, press A to select it, move to another card, and press A to move the first card over to it.

But this is where the classic question arises: Is this the most efficient experience that can be delivered?

The desktop app supports F6 to move between the three main groups in the app. So I updated the app such that a press of the left/right bumpers, (when switch device control is not enabled,) would move focus to the first control in the previous/next group respectively. What's more, given that the access keys on the desktop can provide for some players an extremely efficient means of playing the game, I said that once a bumper is pressed, the keys on the chatpad keyboard would behave in the same way as the access keys on the desktop. For example, press N to turn over a new card, or press U then C to move the upturned card to the Clubs pile, or press 4 then 6 to move a card from the fourth dealt card pile to the sixth dealt card pile.

On the desktop, function keys are used for a variety of actions, and so it was fun to consider how a controller might be used to access that same functionality. In the end, I implemented access to the function key functionality using a mix of buttons on the controller.

By the end of all this, this is how the Sa11ytaire app reacts to input at the Xbox controller:

• D-Pad: Moves keyboard focus left/right/up/down through the app.

• A Button:
  • Invoke the Next Card button.
  • Check the Upturned Card button or Target Card Pile buttons.
  • Select an item in the Dealt Card Pile lists.

   

• LeftThumbstick Button: Replicates A button action, in order for the game to be playable with controls only on the left side of the controller.

• Left/Right Bumpers:
  • If switch device control is not enabled, moves keyboard focus to the first card in the three main areas in the app. (Equivalent to Shift+F6 or F6 on the desktop.)
  • If switch device control is enabled, then triggers the switch device action in the app. (Equivalent to Space key.)

• RightThumbstick Left/Up/Right: Have Narrator announce the state of the remaining cards, target card piles, or dealt card piles respectively. (Equivalent to F2/3/4.)

• RightThumbstick Down: Have Narrator announce the hint on whether a card can be moved between the dealt card pile lists. (Equivalent to F7.)

• RightThumbstick Button: Restart game. (Equivalent to F5.)

• Y Button: Toggle the state of switch device control. (Equivalent to F10.)

• X Button: Deselect and uncheck everything. (Equivalent to Escape.)

Figure 3: The Sa11ytaire app running with Narrator on an Xbox. The four of clubs is being moved onto the five of diamonds.

Summary

I'm really excited to have been able to port the Sa11ytaire experiment to the Xbox. While it was a fun learning experience for me to have published an Xbox game to the Microsoft Store for the first time, what I'm so thrilled about is to now have the opportunity to learn from game players as to what the app really needs to do if it's to provide an efficient experience in practice for players using Narrator or a switch device. And if some of the feedback we get is "You're nowhere near that", then that's exactly what we need to know. And who knows, maybe for some players, we're already pretty close today.

So let us know, and together we can all be a part of delivering that great experience on Xbox.

Guy

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Introduction

A few days ago I was contacted by a developer who had a bug assigned to them, relating to Narrator not announcing a change in state of their checkable menu item. I found the subsequent investigation really interesting, as I don't think I've encountered a situation quite like this before. So I thought it'd be worth sharing the steps I took, in case anyone else discovers a similar bug with their UI.

Spoiler alert: The root cause seems to be due to the behavior of a UI library being hosted by the product. If that really is the case, then it would seem possible that any product hosting this library will exhibit the same accessibility bug.

The UI in question related to a menu in a dropdown. At any given time only one menu item could be checked, and when it was checked, its background changed from white to grey. The following images are my own simulation of the UI. The first image shows a menu containing three items, and the middle item has a grey background, indicating that it's checked, and a black border, indicating that it has keyboard focus.

Figure 1: A dropdown menu containing the three menu items of "Chickadee", "Towhee", and "Grosbeak". The menu descends from a button showing a bird icon.

When the down arrow key is pressed while the menu is in the above state, keyboard focus moves to the next item in the menu, as shown in the following image.

Figure 2: The second menu item remains grey, indicating that it's checked, and keyboard focus has moved to the third menu item.

And finally, if the spacebar is then pressed, the third item becomes checked and the second item unchecked, as shown in the following image.

Figure 3: The second menu item is now unchecked, and the third menu item is now checked and has keyboard focus.

The bug is that when the menu item becomes checked, Narrator says nothing, (other than "Space" if the keyboard echo setting is on). The change in checked state is not announced by Narrator when the state of the menu item changes, and so the customer is not made aware of the change. They'll no longer know what state the UI is in, and that's not acceptable.

So the steps below describe one approach to investigating this bug. It's important to remember that Narrator only cares about the programmatic representation of the UI, as exposed through the UI Automation (UIA) API. The fact that the background of some menu item happens to be grey, means nothing to me in this investigation.

Tip: For a very quick introduction to UIA, check out UIA at a glance. For a more detailed introduction, check out Introduction to UIA: Microsoft's Accessibility API.

Step 1: What state is the UI really in after it was meant to change?

The first thing I'm interested in is whether the checked menu item is really checked from a UIA perspective. Semantically, if a UI element can have a state of checked or unchecked, then it would support the UIA Toggle pattern. By using the Inspect SDK tool, I can verify whether the Toggle pattern's ToggleState value is "On" for the checked menu item, and that it's "Off" for the other items in the menu. Sure enough, the ToggleState values were all as I'd expect them to be, so that's good.

Note that in some cases, we might find a bug like this is due to the checkable UI not supporting the UIA Toggle pattern at all, and we'd need to figure out why it doesn't support the pattern. But that's not the case here.

Figure 4: The Inspect SDK tool reporting that the second menu item has a UIA Value property of "Towhee", and a ToggleState property of "On".

And for completeness here, now that I know that the UI seems to be in the expected programmatic state, I can arrow away from the checked menu item, and back to it, and verify that Narrator does announce the new checked state. When I did that, Narrator's announcement included the new state of the menu item just fine.

Step 2: Remove Narrator from the equation

So, if the Narrator experience is not as it should be, then it must be a Narrator bug right?

Wrong.

Sure, Narrator can be improved, just like any product can, and occasionally some customer experience issue crops up which is caused by Narrator itself. But Narrator's only one of many components that are involved with delivering the customer experience.

This is the stack that I usually care about:

• Narrator, the UIA client app.
• UI Automation itself.
• The UI Framework, which implements the UIA provider API on behalf of the product.
• The UI implemented by the product developer

My first step when investigating the problem with the menu item interaction, is to try to remove Narrator from the equation. If I can do that, then that helps pinpoint where the root cause of the bug may lie.

Narrator is a UIA client app, and so interacts with the product UI of interest through the UIA client API. So if I point another UIA client app at the product UI, and that other UIA client app behaves exactly as expected, then that might suggest the problem lies with Narrator. But if instead, the other UIA client app also struggles at the product UI, then that might suggest I should focus on the product UI, rather than on Narrator.

The bug relates to how Narrator reacts to a change in state of the product UI. In order for narrator to react to a change in the state of UI, the UI must raise a UIA event to make Narrator aware of the change. If no event is raised, then Narrator isn't made aware of the change, and can't make your customer aware of the change. For a one-minute introduction into UIA events, check out Part 4: UIA Change Notifications.

So the next question is: Did the appropriate UIA event get raised when the menu item was checked?

The AccEvent SDK tool is a UIA client app, and can report details of UIA events being raised by UIA. So I'll point the tool at the UI, and check whether a ToggleStatePropertyChanged event is being raised when I select the last item in the menu.

The following image shows what I would have expected to have been reported in AccEvent if everything was working as it should.

Figure 5: The AccEvent SDK tool reporting a UIA ToggleStatePropertyChanged event being raised by a menu item.

However, it turned out that there was no property changed event reported by AccEvent when the menu item was checked. As such, this seemed to suggest that this bug isn't caused by Narrator. Rather the UI isn't making Narrator aware of the change.

Step 3: Why isn't the UIA event being raised?

Now, this is where my investigation became a real learning experience for me.

Traditionally I've found that if AccEvent doesn't report a UIA event, then the event wasn't being raised. So I then ask the product teams for details on exactly how the UI is implemented. In desktop UI, if a standard control is being used from the Win32, WinForms, WPF or UWP XAML frameworks, then a UIA ToggleStatePropertyChanged event would be raised by the UI framework as required. And for web UI hosted in Edge, if the UI is defined using semantic, industry-standard HTML, (including in this case making sure it has a role of "menuitemradio" and uses "aria-checked"), then when an element is checked, Edge would raise the expected UIA ToggleStatePropertyChanged event.

And often during a bug investigation such as this with web UI, we'd examine how the UI was defined and find it didn't use semantic, industry-standard HTML. Rather the UI was built to react visually to customer interaction, but not programmatically. Consequently, the product team would update their UI to use semantic, industry-standard HTML, and the bug would be resolved. Jolly good.

In this particular case the UI was defined in HTML, but by a UI library not created by the product team. So it wasn't quite as straightforward for us to investigate what might be happening.

Step 4: Did the UI change far more than it seemed when its state change?

I'm going to confess that at this point, I was stuck. I flailed around somewhat, trying to think of any way to make progress on this. But after a while, I did discover one critically important piece of information, which at least unblocked the product team. So I'll jump to the useful ending of the story here…

In my experience, if AccEvent doesn't report a UIA event, then the event wasn't raised. (That's assuming I set the AccEvent to report the events of the appropriate type and scope, in its Settings UI.) But what if AccEvent did receive the event from the UI, yet didn't report it? When AccEvent receives an event, it goes back to the sender of the event to learn more about the sender. AccEvent will gather up details of the sender, and display those details in its UI. Perhaps if AccEvent were to have problems gathering up the sender's details after receiving the event, then the event won't get reported?

So, say the menu item raised the expected ToggleStatePropertyChanged event. Having done that, the product then destroys that menu item. It then creates a new menu item, in the checked state, and inserts the new menu item into the menu, in the same place that the previous menu item was. While this is happening, a UIA client app listening for events, (such as AccEvent or Narrator,) receives the event, and returns to the source element to gather details about it. The attempt to get those details fails, because the sender's been destroyed. As such AccEvent and Narrator both discontinue the attempt to react to the event. And given that the new menu item was always checked once inserted into the UIA tree, no ToggleStatePropertyChanged event was raised by that new menu item.

That hypothesis seemed rather unlikely to say the least, but it would match the results that we were experiencing.

So to pursue that line of thought further, we needed to know whether the UIA element representing the menu item of interest was the same UIA element both before and after the change in checked state. Comparing UIA properties such as the Name or AutomationId wouldn't help here, as they're very likely to be the unchanged throughout. But if the UIA RuntimeId property has changed, then it's not the same element. (I mentioned something about RuntimeIds a while back, at Don't use the UIA RuntimeId property in your Find condition.)

So I pointed the Inspect SDK tool at the menu item when the item was not checked, and then again once it had become checked. And having done so, I found the RuntimeIds were different. For example, in one run-through, the RuntimeId changed from "2A.80A44.4.816C", to "2A.80A44.4.81C5".

Figure 6: The Inspect SDK tool reporting the UIA RuntimeId property for a menu item.

At this point, it seemed that we knew enough to follow up with the owners of the library which created the menu UI. I strongly suspect that the action being taken to replace the menu item UI during the interaction, is leading to UIA clients like Narrator not being able to make the customer aware of the change in state at the time the change happens. Whether the ultimate resolution is to have the library updated to not recreate UI in this way during customer interaction, or to replace the use of the library with semantic, industry-standard HTML which simply checks or unchecks an item which lives through the interaction, I don't know. But at least the product team is unblocked.

Summary

This investigation has been a reminder of a couple of things:

• The visual representation of UI is no indication of the programmatic representation. Our customers require both representations to be a good match for the meaning of the UI, but depending on how the UI is implemented, that requirement might not be met by default, particularly for web UI. So be sure you're familiar with both representations.

   

• Before considering leveraging any library to present UI in your product, where the action taken by the library is outside of your control, always be sure that the UI that the library provides is fully accessible. Whether that UI is a menu item or a full-blown complex chart, you don't want to risk shipping UI that's inaccessible. Your customers won't care that it's some library UI hosted in your product that's blocking them. All they care about is that they can't use your product.

   

And by the way, while this particular investigation has involved web UI, the principles apply to any UI. For example, say you want your UWP XAML app to make your customers that are using a screen reader aware of some important change in the app's UI. You call FrameworkElementAutomationPeer.FromElement() to get the AutomationPeer associated with a control, then call that AutomationPeer's RaisePropertyChangedEvent(), and then for some reason destroy the control. If Narrator or any other UIA client quickly comes back to the app to learn more about the UI that raised the event, it's not going to be able to do much that's helpful to your customer if that UI's already been destroyed.

'Til next time.

Guy

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Introduction

As we played more with the Sa11ytaire app after its initial release, and got feedback from others, we learned how we could provide a more intuitive and efficient experience for players using a screen reader such as Narrator. Narrator interacts with the app solely through its programmatic representation as exposed through the UI Automation (UIA) API, and so we set to work on enhancing that programmatic representation of the app, as described in the Three Updates section below.

Actually, we didn't. All the updates were made by my co-developer Tim, so I'm just describing his work here. So a big thanks to Tim for the work he put in for these improvements to the app!

A couple of additional thoughts

Thought #1

We did get some very positive feedback relating to the access keys that we'd originally added to the app. For some players, pressing the Alt key followed by some other key, can be a very efficient way to play the game. The image below shows the access keys being revealed visually in the app in response to a press and release of the Alt key. The interactable elements shown along the top of the app have access keys of N, U, C, D, H and S. The seven dealt card piles have access keys of 1 through 7.

Figure 1: Access keys being shown visually in the Sa11ytaire app.

The UWP XAML framework often makes it extremely quick 'n' easy to implement access keys. All the interactable elements along the top of the app are implemented using Buttons and ToggleButtons, and when the player presses the access keys, the associated Button's Click handler, or ToggleButton's Checked handler, is called. So that's some really powerful functionality added to the app, simply by adding something like:

AccessKey="N"

The access keys for the dealt card piles are a little more interesting. I expect different apps might need to react differently in response to the press of a ListView's access key. The Sa11ytaire app needs keyboard focus to move to the last item in the list, and for that item to be selected. In order to achieve this, all the dealt card ListViews have the following markup applied:

AccessKeyInvoked="CardPile_AccessKeyInvoked"

When that AccessKeyInvoked function is called in response to the press of the access key, it moves keyboard focus and selects the item, using the steps shown below.

private void CardPile_AccessKeyInvoked(UIElement sender, AccessKeyInvokedEventArgs args)

{

    ListView list = sender as ListView;

    if (list != null)

    {

        if (list.Items.Count > 0)

        {

            list.Focus(FocusState.Keyboard);

            list.SelectedIndex = list.Items.Count - 1;

        }

    }

}

To learn more about access keys in UWP apps, visit Access keys.

While overall this design works great, we did learn of one unexpected problem with the interaction model that relates to the discoverability of some of the access keys. An app might have all sorts of handy features which can lead to an efficient experience, but if the player isn't aware of the features, then the features might as well not exist. So the player must be made aware of all the access keys available in the Sa11ytaire app. While the game's Help documentation describes all available access keys, wherever possible, the player shouldn't have to refer to the Help documentation. Instead we want it to be intuitive to the player as to how to play the game efficiently.

So consider the player using the Narrator screen reader with the app. Say they tab to move keyboard focus to the button called "Next card". Narrator examines the UIA properties for the button, and learns that the UIA AccessKey property is "Alt, N". Narrator can then include that information in its announcement, and the player is made aware of this efficient means to invoke the "Next card" button. Going forward, the player doesn't need to tab to the button, instead they simply press Alt+N. Brilliant!

However, say the player later tabs to a card in a dealt card pile. The items in the dealt card piles don't have access keys, rather it's the ListViews that contain the items that have the access keys. So when keyboard focus moves to an item in the list, Narrator doesn't announce the access keys. As such, the player is not made aware of the really efficient means to move to, and select, an item in a list of dealt cards. At some later time the player may read through the app's Help documentation and learn of the access keys for the lists, but it really would be preferable for them to not have to do that.

The image below shows the Inspect SDK tool reporting the UIA representation of one of the dealt card ListViews.

Figure 2: The Inspect SDK tool reporting the UIA representation of one of the dealt card ListViews, with the ListView's UIA AccessKey property of the "Alt, 2" highlighted in the image.

We're still figuring out the most practical way for us to improve this experience, such that the player is automatically made aware of all the access keys that are available in the game. Narrator's Context Verbosity setting can be configured to have some information relating to an element's parent element announced when Narrator reaches the element. For example, when tabbing from the second to the third card pile list, Narrator's announcement can include "Exit Card pile 2 list, Enter Card pile 3 list". But the Context Verbosity feature can't trigger inclusion of the parent's AccessKey property when tabbing to an item in the list. And even if it could, we'd never base our actions on the Narrator experience alone. Instead, we focus on shipping the best programmatic representation of the game that we can through UIA, and have that representation be leveraged by all assistive technology apps however those apps feel is most helpful to your customers.

Perhaps one practical approach here would be for us to manually set the AccessKey property of the last item in each list. The lists already all have LayoutUpdated handlers, which are used to set the height of the items in the list. That ensures that the last item in the list is a lot taller than the other items in the list. So as a test, I just updated an existing function in the app which is called beneath the LayoutUpdated handlers, such that it also sets the UIA AccessKey properties of the items in the list. It goes something like this:

private void SetHeightOfCards(ListView list)

{

    if (list != null)

    {

        for (int i = 0; i < list.Items.Count; ++i)

        {

            var card = list.Items[i] as PlayingCard;

            var item = (list.ContainerFromItem(card) as ListViewItem);

            // Is this the last item in the list?

            if (i == list.Items.Count - 1)

            {

                item.Height = 120;

                // Give the last item in this list the same access key as the list itself.

                string listAccessKey = list.AccessKey;

                AutomationProperties.SetAccessKey(item, "Alt, " + listAccessKey);

            }

            else

            {

                item.Height = 60;

                // This list item has no access key.

                AutomationProperties.SetAccessKey(item, "");

            }

        }

    }

}

With the above code, when Narrator reaches the last item in the list, it announces the access key used to move focus to the item and to select it. It really doesn't matter from the player's perspective that the UWP XAML AccessKey is actually implemented on the containing list rather than the list item. The following is an example of the announcement made by Narrator with this change, as I tab to a card in a dealt card list.

"Card pile 3, 10 of Hearts, 3 of 3, non-selected, Alt, 3"

I don't know if there's a way to achieve the same results through binding, but all in all the above approach was pretty straightforward and seemed quite useful. If it holds up after more testing, maybe we'll include it in the next update of the app.

Thought #2

When Microsoft recently hosted Benetech's 2018 DIAGRAM Code Sprint, I had an opportunity to demo the Sa11ytaire to a couple of attendees and get their feedback. As I was demo'ing the switch device control of the app, I was asked whether the same input model would work with other card games. I'd not really thought about that before, but I expect it would work great with a number of other card games. From memory card games to poker, the first step would be to figure out the most efficient path for the player to reach the UI elements they're interested in.

On a side note, I'm particularly interested in the idea of an accessible slider puzzle app. Way back I spent lots of time playing with slider puzzles, and Sa11ytaire's switch device control would seem to be a good match for such puzzles. Particularly given that at any given time there are only a few moves that can be made in the puzzle.

I also got a question about how the Sa11ytaire app might be updated for players using multiple switches, and it was suggested that different switches could initiate the scan through the UI at different areas in the app. Perhaps I could take the opportunity to explore that idea with the new Xbox Adaptive Controller. Today the Sa11ytaire app's switch control reacts to a press of the Space key, which means the switch device must be configured to simulate that. But I expect it'd be straightforward to update the app to react to different types of input coming from a game controller. That'd certainly be a fun experiment.

And on a rather different note, I can't help but wonder if it would be possible to effectively add switch control support to an app, from outside the app. If the target app had a good UIA representation, then its UIA tree could be manually examined by a dev to determine the most efficient path for the switch scan highlight to move through the app's UI. A standalone UIA client app could access the bounding rects of these elements, and draw a highlighting rectangle above the app, in a similar way to how my Herbi HocusFocus app highlights where keyboard focus is. Once the switch scan highlight is above the element of interest in the target app, a press of the switch would trigger either a change in level of the scan, (for example, moving into a list to scan through the items,) or perform some action on an element through a UIA pattern, (for example, invoke a button or select a list item). I doubt I'll ever have time to try that out, but it would seem technically possible.

I am grateful for the feedback I got at the recent Benetech event, and if we do get requests from people considering using the app with a switch device, I'll look forward to learning how we can make the app as efficient to use as possible.

Figure 3: A switch device being used to play the Sa11ytaire game.

Three updates

The following sections describe the UIA-related updates made to the Sa11ytaire app.

Landmarks

A quality app provides an intuitive means for the customer to navigate around the app's UI in an efficient manner. And "intuitive" here means that if the player is familiar with standard keyboard shortcuts that work in many other apps, then they'll try using them in the Sa11ytaire app.

One common keyboard shortcut for moving keyboard focus between areas in an app, is F6. Try pressing F6 in products like PowerPoint, Outlook, and Windows Explorer, and consider how keyboard focus is moving around the UI. F6 provides a quick way to move keyboard focus into some area of interest in the app, after which the Tab key might be pressed a few times to reach a control contained inside that area. And of course, for commonly accessed controls and functionality, we'd consider adding access keys or accelerator keys to make the steps to complete a task even more efficient.

So when we first built Sa11ytaire, we did add support for a press of F6 to move between the three main areas in the app. With a press of F6, keyboard focus would move between the Next card button, the target pile for clubs, and the last item in the first dealt card pile.

While F6 can help to provide a more efficient experience for all players using the keyboard, there is also a very well-known means of efficient navigation specific to using a screen reader like Narrator. And that is to navigate between "landmarks" in an app's UI. The concept of landmarks originally sprang from the web, and some related details for web UI are available at Using ARIA landmarks to identify regions of a page. But these days it's also quick 'n' easy for a dev to add landmarks to a UWP XAML app.

For example, let's consider the Microsoft Store app. That app presents UI for navigating around various pages in the app, and that UI is always available near the top of the app, regardless of where the customer happens to be in the app. I just pointed the Inspect SDK tool at the UI, such that I can examine the app's UIA representation, and I find that a container for that navigation-related UI has a UIA LandmarkType property of UIA_NavigationLandmarkTypeId. This lets any UIA client app like Narrator know that the container is a landmark, and its purpose relates to navigation through the UI. There's an additional LocalizedLandmarkType property which provides a localized string to the UIA client app, so that the landmark's purpose can be presented to the customer in a helpful human-readable way.

Figure 4: The Inspect SDK tool reporting the UIA properties of a landmark in the Microsoft Store app, with the landmark-related properties highlighted.

If I go on to explore the Store app's UI further, I find another landmark in the UI, and its LandmarkType is UIA_MainLandmarkTypeId. That signifies where the primary content is on the page. (Exactly what the "primary content" is, is up to the app dev to decide.) Yet another landmark in the Store app's UI has a LandmarkType of UIA_SearchLandmarkTypeId, and it's associated with the UI for initiating a search in the Store.

Once an app exposes a set of useful landmarks in its UI, a customer using a screen reader such as Narrator can issue commands to quickly move between the landmarks. In Narrator's case I can press CapsLock+Space to turn on its Scan mode, and then press D or Shift+D to move to the next or the previous landmark. For example, if I press Shift+D and move backwards in the Store app to the navigation area, Narrator's announcement lets me know that I've moved to the "Home" item, and that it's contained within the navigation landmark. The announcement is as follows:

"Home selected, selection contains 8 items, navigation landmark"

The UWP XAML framework makes it really straightforward for an app like the Microsoft Store app to support landmarks. For example, to make some container element in the UI the Main landmark, add the following markup:

AutomationProperties.LandmarkType="Main"

So how might landmarks be leveraged by the Sa11ytaire app? The app has three areas which would seem good candidates for landmarks, enabling the player to move quickly through them using a familiar navigation technique. Those areas are the remaining cards, the target card piles, and the dealt card piles. But none of those seem a good match for a "navigation" landmark or a "main" landmark. In fact, they seem quite specific to the app. As such, we gave the containers associated with those areas a LandmarkType of UIA_CustomLandmarkTypeId. This indicates that we want the elements to be accessible through screen readers' landmark-navigation functionality, but the type of landmark is specific to the app.

While that alone provides some useful navigation functionality, it doesn't provide the best experience for the player, in that the player isn't immediately made aware of the purpose of the landmark. By default, UIA will provide a LocalizedLandmarkType of (the localized form of) "custom". That might not seem particularly descriptive, but what else can UIA do? UIA doesn't know the meaning of the UI like we do. Indeed, we could set some far more helpful text through the UIA LocalizedLandmarkType property.

Setting a LocalizedLandmarkType property is as quick to do in a UWP XAML app as setting any other localized string on an element. First we make sure the element has a x:Uid, and then we add an associated AutomationProperties.LocalizedLandmarkType in the localized string resource file. By doing this, we provided LocalizedLandmarkType properties of "Remaining card piles", "Target Card Piles", and "Dealt card piles".

The following is the announcement made by Narrator when it reaches the first area in the app following landmark navigation.

"group, Remaining card piles landmark"

Figure 5: The Inspect tool reporting the UIA properties of the group element associated with the first area in the app. The landmark-related properties are highlighted.

Considering the UIA representation of the landmark elements further, it would seem appropriate for us to update their UIA representation to also give them UIA Name properties. The landmark-related elements in the app are all XAML StackPanels, which got exposed through UIA as elements with ControlTypes of Group. But we didn't set an AutomationProperties.Name on those StackPanels. During implementation of the landmarks, it seemed sufficient for the elements to have appropriate ControlsTypes, LandmarkTypes and LocalizedLandmarkTypes, but really, if an element's important to the player, it should have a helpful UIA Name too. We'll consider this and determine what the most helpful Names would be. Simply replicating the LocalizedLandmarkType in the Name might not lead to the best experience for the player.

And on a similar note, Figure 4 above shows Inspect reporting that the navigation landmark-related list in the Store app doesn't have an accessible name. By default all lists want UIA Names, so that anyone moving into a list and reaching some item can be made aware of exactly which list they're in.

One more note: it seems that the player experience is not quite as we expected when Narrator interacts with one of the landmarks in the app. Typically when the player has used landmark navigation to reach a landmark, they can press the Tab key to move keyboard focus to the first focusable element contained in the landmark. While that works fine for the first two landmarks in the app, it doesn't work at the third landmark containing the dealt card lists. Instead, a press of the Tab key after navigating to that landmark actually moves keyboard focus beyond the lists. It doesn't look like we can do anything about this, and like I said earlier, we'd never change the app's UIA representation solely to account for how Narrator behaves today. So this means once a player using Narrator has navigated to the dealt card list area via landmark navigation, they'd use Narrator's item navigation functionality to move into the first dealt card list.

Notifications

Recent versions of Windows 10 support UIA's UiaRaiseNotificationEvent() function. This enables an app to raise a UIA Notification event, and by doing so, request that a screen reader announce any arbitrary string that the app feels is helpful to your customer. I discussed some of the ways that different types of UI can raise this event, (that is, Win32, WinForms, WPF and UWP XAML,) at Can your desktop app leverage the new UIA Notification event in order to have Narrator say exactly what your customers need? And for UWP XAML apps, it's really straightforward to raise the event, by calling AutomationPeer.RaiseNotificationEvent().

Warning: Just because it's easy to raise the Notification event, doesn't mean that apps should raise it. An app needs to strike a balance between making sure its customers are made aware of everything important that's going on in the app's UI, while not flooding the customers with unwanted announcements which are at best a distraction and at worse block the customers from completing their tasks. So only raise the Notification event if that will really help the customer experience in practice.

When Sa11ytaire first shipped, we found that the notification-related experience for the player wasn't always as we'd hoped. We experimented with the various AutomationNotificationProcessing values that we could pass into AutomationPeer.RaiseNotificationEvent(), and found that announcements could be interrupted in a way that didn't seem to match what we'd requested. But in the Windows 10 April 2018 Update, we're finding the experience more as we'd expect, and so we revisited how we call the RaiseNotificationEvent () function.

With the first release of Sa11ytaire, we'd always pass in ImportantAll to RaiseNotificationEvent(). For the latest release of the app, we've changed some of the calls to use ImportantMostRecent instead. For example, consider when a card is moved from a dealt card pile to a target card pile. When that happens, keyboard focus first moves to the card element in the list, and then moves to the card element in the target card pile. By default Narrator will follow keyboard focus as this happens, and make an announcement such as:

"Ace of Diamonds, 5 of 5, selected, Alt, 5"

"off, Ace of Diamonds, button, Alt, D"

However, we want the app to also make an announcement relating to the card that's been revealed in the dealt card pile following the card being moved. Keyboard focus is not at that revealed card, and so Narrator won't announce it by default. So we raise a Notification event to have the revealed card announced. By raising the event with an AutomationNotificationProcessing value of ImportantAll, then the announcement will be made in its entirety regardless of whether some other event happens to be raised by the app around the same time. The announcement will not be interrupted by another announcement. This means that in addition to the example announcements shown above, the player will also be made aware of (say):

"Revealed 5 of Diamonds in dealt card pile 5."

Importantly this means that if the FocusChanged event that's being raised by the app in response to keyboard focus moving, is received by Narrator after Narrator receives the Notification event, then the announcement relating to the focus changing won't interrupt the announcement relating to the card being revealed. Before the introduction of the UIA Notification event, this Narrator experience couldn't always be reliably delivered by a UWP XAML app.

We also use Notification events in the implementation of the feature where the player can request information about the three main areas in the app. At any time, the player can press F2, F3, or F4, to learn about the state of the remaining cards, the target card piles, and the dealt card piles respectively. For example, in response to a press of F4, Narrator might announce the following.

"Pile 1: Jack of Spades to King of Spades, Pile 2: Jack of Hearts to Queen of Clubs, 1 card face-down, Pile 3: 8 of Spades, 1 card face-down, Pile 4: 4 of Diamonds to 5 of Clubs, 3 cards face-down, Pile 5: 5 of Diamonds, 3 cards face-down, Pile 6: Jack of Clubs, Pile 7: 9 of Spades to 10 of Diamonds, 5 cards face-down"

It's quite possible that early in that announcement, the player will be made aware of some information which they want to act on immediately. For example, the player might feel there's potential for a card to be moved to a target card pile. As such, part way through the above announcement, they may press F3 to learn of the state of the target card piles. In that situation, the announcement relating to the target card piles should occur immediately, and so interrupt the in-progress announcement relating to the dealt card piles. It would be really irritating for the player if they had to wait for the dealt card pile announcement to complete before learning of the state of the target card piles. As such, in the new release of Sa11ytaire, when we call AutomationPeer.RaiseNotificationEvent() in response to a press of F2, F3 or F4, we pass in an AutomationNotificationProcessing of ImportantMostRecent.

This new experience seems to be holding up fairly well with the new release Sa11ytaire on the Windows 10 April 2018 Update, and we're looking forward to learning what more we can do with the Notification event in the future to further enhance the player experience. See AutomationNotificationProcessing for the full list of values that can be passed in to the call to RaiseNotificationEvent().

Control Types

The third change to the app is not so much adding helpful new functionality, but rather it's fixing something that wasn't quite right in the first release of the app.

While all the interactable controls in the Sa11ytaire app are based on standard controls, such as Buttons and ToggleButtons, they have been customized to some extent. When I first did this, I made sure the UIA Name property reflected the suit and value of the card shown visually. For example, "Jack of Hearts". By doing this, Narrator would announce that suit and value when it encountered the card, and a player using Windows Speech Recognition could say "Jack of Hearts" to interact with the card.

But at the time, I also customized the LocalizedControlType to provide some more descriptive information about the card. For example, I'd given the top three cards in the upturned card piles, LocalizedControlTypes of "Top upturned card", "Second from top upturned card", and "Third from top upturned card". But Tim later pointed out that really, those strings aren't conveying control types. The strings gave no indication to the player as to how they can interact with the card, and an element's UIA LocalizedControlType property should wherever possible set the player's expectations around the functionality available on the control and how they can interact with it.

So the Sa11ytaire app was changed to have the default ControlTypes and LocalizedControlTypes exposed on the various elements in the UI. The screenshot below shows the Inspect SDK tool reporting that the top upturned card is now exposed as a Button which supports the UIA Toggle pattern. As such, the player is informed that they can interact with the element in the same way as any other toggleable button in a UWP XAML app.

It's still important to convey the information that was previously exposed through the LocalizedControlType, so that's now been moved to the UIA HelpText for the card elements. With this change, the following announcements are made as Narrator's scan mode is used to navigate from the remaining cards UI through the subsequent UI in the app.

"Next card, button, Alt, N, Single pile of cards, face down, Invoke this to move the top three cards from this pile, to the pile of upturned cards"

"off, Queen of Hearts, button, disabled, Third from top upturned card"

"off, 3 of Hearts, button, disabled, Second from top upturned card"

"off, 6 of Hearts, button, Alt, U, Top most upturned card"

This would seem to lead to a more intuitive experience for the player, and it's been a reminder for me of how I need to be really careful when I override the default UIA experience for the game's UI. Sometimes the default UIA properties are actually what the player needs.

s

Figure 6: The Inspect SDK tool reporting the UIA properties of an upturned card element, with properties relating to Name, ControlType, HelpText, and the Toggle pattern highlighted.

Summary

With each new release of Windows, the support for accessibility in UWP XAML apps continues to grow. I'm really pleased that Tim and I have been able to take advantage of some recent enhancements such as use of landmarks and more well-behaved notifications. This post certainly doesn't describe all the recent platform enhancements, so you may find that your apps can leverage exciting new accessibility-related functionality that the Sa11ytaire app doesn't. For example, a UWP XAML app can now define heading levels for text presented in its UI. Use of headings could dramatically improve the experience for customers using screen readers in some types of apps.

Please keep the feedback on your experiences with the Sa11ytaire app coming. Whether the feedback relates to using a screen reader, a switch device, eye control, or some other form of input, we always value your thoughts on how we can make the app more intuitive and efficient to use.

All the best,

Guy

Apology up-front: When I uploaded this post to the blog site, the contained image did not get uploaded with the alt text that I'd set on it. The image is followed by a textual description of the image.

Introduction

I got a reminder recently of how important it is for app devs who are using the Windows UIA Client API or Windows UIA Provider API, to be aware of the threading model being used by their app. I was asked by some devs who'd built their own UIA provider, about unexpected delays being experienced when a screen reader was interacting with their app. As it happened, I didn't know the cause of the problem, but they figured it out. It related to the threading model being used by the app. So this was a helpful learning experience for me, and a great reminder to always keep the threading model in mind.

The following are a few thoughts on threading considerations when using the Windows UIA API.

The UIA Client app

Many assistive technology (AT) apps use the Windows UIA Client API, including the Narrator screen reader, Windows Magnifier, and the Windows On-Screen Keyboard.

Any UIA Client app can experience unexpected delays beneath calls into UIA, if the threading model is not considered. Some very important points are discussed at Understanding Threading Issues. For example, making sure CoInitializeEx() is called with COINIT_MULTITHREADED if required.

To give an example of where an inappropriate threading model can be problematic, I once worked on a single-threaded UIA client app which, called CoInitialize(NULL) on startup. It then called AddFocusChangedEventHandler(), (remember, that's being called on the app's UI thread,) to register for notifications as keyboard focus moved around the system. Importantly, in that call to AddFocusChangedEventHandler(), it passed in a request to have the UIA BoundingRectangle property of the element gaining keyboard focus cached with the event. This meant that when handling the event, the app could call get_CachedBoundingRectangle() and get the bounding rect without an additional cross-proc call being made back into the UIA provider. Having got the bounding rect of the element with keyboard focus, the app could react however it pleased, and all seemed well.

Well, yes, all did seem well until the call to get_CachedBoundingRectangle() was replaced with a call to get_CurrentBoundingRectangle(). For some reason, it seemed necessary in some scenario to call back into a provider to get the bounding rect of the element with keyboard focus after the event had been received. (Perhaps the provider hadn't updated the element's UIA BoundingRectangle property until after it raised the event saying the element had got keyboard focus, and so whatever bounding rect was cached with the event, was stale.) But once the call was made in the UIA client to go back to the provider, that was happening for any element getting keyboard focus, including an element in the UI of the UIA client app itself. With this change, long delays intermittently starting being hit beneath the call to get_CurrentBoundingRectangle().

So really, if the app is expected to be able to handle a UIA event from any UI, (including its own UI,) and expected to be able to call back into providers while inside event handlers, it was on thin ice going with a single-threaded approach. If the app wants to be reliable, it'd spin up a background thread, initialized with CoInitializeEx(NULL, COINIT_MULTITHREADED). Inside that thread, it'd CoCreateInstance() a CLSID_CUIAutomation8, and call AddFocusChangedEventHandler() and RemoveFocusChangedEventHandler() on that thread. Whatever UIA calls it needs to make in the UIA event handler would be made on that background thread. And if some related action ends up needing to be taken on the UI thread, a Win32 app might post a custom message to that thread, or a WinForms app might call BeginInvoke(), to make that happen. And all would really be well.

Back in the Windows 7 days, I created a set of UIA client samples at Developer Code Samples. My goal was to demonstrate use of some of the most common actions a UIA client app might take. I really worked to feel confident that there was no chance that the threading model made the sample less robust. (In fact a good chunk of the sample code relates to threading, and not really to the UIA client calls.) I think it's possible that since Windows 7, the threading constraints around UIA have been relaxed a little, and so perhaps not all the threading action in my samples is still necessary, but I don't know the exact details if that is the case. I believe that at least some of the action I took must still be accounted for. For example, as Understanding Threading Issues states, calls to RemoveAutomationEventHandler() must be made on the same thread as the matching AddAutomationEventHandler() was called on, and they must be called in a non-UI Multithreaded Apartment (MTA) thread. So if you start calling either of those in a UIA event handler, are you sure that you're accounting for those constraints? For me, I'd never call either AddAutomationEventHandler() or RemoveAutomationEventHandler() in a UIA event handler. (Sometimes it can be tempting to want to do that, in order to add property changed handlers to an element, whilst inside the event handler that's letting you know the element's got keyboard focus.)

The UIA Provider app

I've less experience with implementing the UIA Provider API than I have with the UIA Client API. This is because, like most other devs, I let the UI framework that I'm using implement the UIA Provider API for me. For example, whether I'm building UI using Win32, WinForms, WPF, UWP XAML, or web UI hosted in Edge, I use standard controls and widgets, and let the UI framework take what I've built and generate the appropriate UIA representation for me. (Of course, I still take UI framework-specific steps which will enhance that default UIA representation if my customers need me to.) But there are still some very important scenarios where devs choose to implement the UIA Provider API themselves.

So I was contacted recently by the devs experiencing delays when a screen reader interacted with their custom UIA provider. One point in the description of the issue seemed particularly interesting; the delays went away when they changed their implementation of IRawElementProviderSimple::ProviderOptions() to not return ProviderOptions_UseComThreading. However, without that flag, the app started experiencing crashes. According to ProviderOptions, the ProviderOptions_UseComThreading option affects what thread the provider's methods are called on, and that's affected by the type of apartment in which the threads are running. For threads running in a Single-Threaded Apartment (STA), COM will synchronize the calls into the provider, to protect against multiple methods in the provider running concurrently when the provider's not designed to support that. But this can mean that UIA will be blocked from calling into your provider while the provider's busy doing work elsewhere on the same thread. Perhaps this was related to the unexpected delays, and it would be necessary to examine exactly what the thread of interest was doing while UIA was waiting to call the provider.

As it happens, the devs came back to me really quick, saying that they'd replaced the call that they were making to CoInitialize(NULL) with a call to CoInitializeEx(NULL, COINIT_MULTITHREADED), and the delays had gone away. In their case the delays that they'd encountered related to the time it took the UIA events that they raised to propagate through UIA to the UIA client app, rather than delays in their methods being called by UIA. But the threading model's going to affect a ton of what goes on inside UIA.

So this really was a learning experience for me. I'd seen the use of CoInitialize() in a UIA client app lead to delays, but I now know that that's one of the first things to be considered in a UIA provider too. With the change to initialize the provider such that it's running in a thread in an MTA, and by having its IRawElementProviderSimple::ProviderOptions() return ProviderOptions_UseComThreading, COM will not block the UIA calls into the provider's methods.

Important: The above approach does mean that the provider had better be thread-safe. If COM is going to happily allow concurrent UIA calls into the provider's various methods, then the provider needs to be designed to handle that. In this scenario, it's the provider's responsibility to implement whatever thread synchronization is required to account for concurrent access.

Summary

The apartments in which the threads are running in a UIA client or UIA provider app are rarely at the forefront of a dev's mind when building the app. Rather, thought is typically more focused on all the ways in which you can use UIA to help your customers. But making sure the appropriate threading model is used in your app is critically important to delivering a robust experience.

A long time ago I wondered how I could encourage devs to make sure that their threading model was included in the list of things they considered when designing their apps. I'd seen severe delays being investigated just before products were about to ship, and it was a serious blow for a dev to learn that the only way that they could make the app reliable was to change the threading model. No-one should be doing that just before shipping. So I created the cartoon below in the hope that it might grab devs' attention and prevent at least some devs from discovering the problem just before shipping. Like so much of what I've done, it seemed like a good idea at the time.

It's almost midnight… DO YOU KNOW WHERE YOUR THREADS ARE?

Guy

The above cartoon contains eleven frames, and is titled "Proxy & Stub Present... Live at the Inproc". At the top of the cartoon, above the contained frames, the character called "Proxy" is standing behind a microphone, saying "A funny thing happened on the way to the apartment", and seems to be wearing historical attire. The character called "Stub" looks on, saying "Nice threads!".

Attached to one of the frames at the top of the cartoon are the lollipop symbols used in technical diagrams to represent interfaces supported by a component. One of the symbols is labelled "IGBarker".

The images in the first ten frames are identical, with Proxy and Stub not looking at the reader, and having the following conversation.

Proxy: We're going to talk about a couple of COM threading models that you can use when developing client applications or server objects.

Stub: Threading Model? What's that?

Proxy: You can think of a threading model as a way for you to control how your client and server objects will work with each other when you have more than one thread running.

Stub: That may be fascinating, but why do I care?

Proxy: Because it may be easy to create an application with lots of threads, but if you don't understand how your objects work across those thread boundaries, you're jeopardizing stability and performance.

Stub: So if I don't design it right up front, I'll regret it later?

Proxy: Right! The first thing to know is that a thread runs in an "apartment". How an object in one thread works with objects in other threads depends which apartments those other threads are in.

Stub: Are there different types of apartments?

Proxy: Yes, you can create your thread in a "Single Threaded Apartment" (STA), and you can have any number of STAs. Or you can create your thread in a "Multithreaded Apartment" (MTA).

Stub: ...in which case it's not an empty MTA eh?

Proxy: Well said. There's at most one MTA in a process, and any object created in the MTA can be accessed by all threads running in that MTA at the same time.

Stub: So those objects better provide synchronization to make them thread-safe?

Proxy: Right! If the object's not thread-safe, then it must run in an STA where only one thread can run. This means that COM will synchronize all calls to the object from all other threads automatically.

Frame notes: Musical notes appear in the background.

Stub: STA, All my troubles seem so far away...

Proxy: Not quite. It takes time for COM to do all this work, and you still have extra work if you're going to pass interface pointers around from one STA to another.

Stub: So I'm better off with MTA?

Proxy: Performance will probably be better with MTA, because you know exactly how and where your objects need to be synchronized.

Stub: HASTA LA VISTA STA! I'm going MTA!

Proxy: The point is you should understand what threads your objects use and which apartments they live in. This will improve stability and performance...

Stub: ...and you'll know what to expect when external components work with yours.

Frame notes: The background of the frame darkens, and both Proxy and Stub look at the reader. In unison they say, "It's almost midnight... DO YOU KNOW WHERE YOUR THREADS ARE?"

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Introduction

Some time back, an organization which works with people with low vision asked me whether it would be practical for me to build a tool which provides customizable feedback to indicate where keyboard focus is, and where the text insertion point is. The result is Herbi HocusFocus at the Microsoft Store, and I described the app's journey in the following posts:

• A real-world example of quickly building a simple assistive technology app with Windows UI Automation.
• Considerations around the accessibility of a WinForms Store app.
• Sharing assistive technology through the Microsoft Store

(Note that for the rest of this post, I'm going to refer to the "text insertion point" as the caret, given that in some apps, the caret can appear in read-only text, and as such, isn't actually a text insertion point.)

The organization got back to me recently to let me know that some of their clients would like a line shown underneath the sentence that they're reading, and wondered whether it would be practical for me to update my app to have that line shown. This seemed like the sort of feature where in some high-profile target apps, the Windows UI Automation (UIA) API could really help. So I spent a few hours adding a first version of the feature to the app, and uploaded it to the Microsoft Store. I knew there'd be certain constraints with what I'd built, but I was hoping it would work well enough to generate feedback, and help me prioritize my next steps.

Certainly the initial response has been encouraging, as I've been told that the update is receiving very positive reactions. From the organization:

"Awesome! I went and tried the feature right away and it's certainly a very great addition for people with cerebral visual impairment and people who suffer from acquired brain impairment/injury."

One handy aspect of the new feature is that the visuals shown for the underline were very straightforward for me to implement. While the other custom visuals shown by the app involve dealing with transparency, this new feature simply moves a window around. The window's positioned below the current line in the text, and is as wide as the line. It has a fixed short height, and has a background of the same customizable color as the caret highlight. Given how straightforward that was, it's really only the UIA interaction that's of interest here.

This has been a reminder for me of how a feature that's relatively straightforward to add using UIA, can have an important impact on people's lives. As such, I'd encourage all devs to consider how you can leverage UIA to add exciting features to your own apps which could help your customers in new ways.

So what are the constraints in my new feature?

Before describing how I leveraged UIA to add the new feature, it's worth calling out a few scenarios which won't be impacted by my recent work. Depending on the feedback I get, I can consider which of these constraints are most impactful to people using my app, and so consider what I can do about that.

The UIA Text pattern needs to be available

In order for my app to learn where the current line is, the target app needs to support the UIA Text pattern. The Text pattern provides a great deal of information about the text shown in an app, and I talked about the Text pattern in the series of posts starting at So how will you help people work with text? However, not all apps support the Text pattern. So if the target app doesn't support the Text pattern, then there'll be no underline shown in the app.

The Inspect SDK tool can report whether a text-related part of an app's UI supports the Text pattern. If the Text pattern is supported, then the element's IsTextPatternAvailable property is true.

Figure 1: The Inspect SDK tool reporting that the editable area in WordPad supports the UIA Text pattern.

The app only works with editable text

The app tracks changes to the caret position in an app, and so won't work in apps where there is no caret. This will typically be the case with apps that show read-only text, and provide no way to move the caret with the keyboard in order to select text.

Checking the UIA ControlType of the element

I wanted to ease into the new feature somewhat, so I could feel confident that things were working as required in a few well-known scenarios, before opening it up to work in as many places where it might work. As such, I checked the ControlType of the UIA element that claims to support the Text pattern, and decided to only show the line beneath the current line of text if the ControlType was Document or Edit. I expect I'll remove this constraint at some point.

You might be wondering what types of control other than Document or Edit would raise a UIA TextSelectionChanged event? Well, say an app presents read-only text, but provides a way to move the caret through the text with the keyboard, in order to select text. I've found one app which does this, and the ControlType of the element supporting the Text pattern is Pane.

A regrettable use of legacy technology

When I first added the feature of highlighting the caret position a few years ago, I made a poor choice. While all the focus-tracking action I took used only UIA, the caret tracking used a mix of UIA, and some legacy technology called winevents. I only included some use of a winevent because I was familiar with how that related to the caret position, and it seemed convenient for me at the time. I did it despite knowing how winevents are desktop-only technology, and so my feature wouldn't work if in the future it's downloaded through the Store to other platforms. And I did it despite UIA supporting a TextSelectionChanged event, which an app can raise whenever the caret moves. Well, I'm regretting doing that now.

It seems I've found a situation where an app raises the UIA TextSelectionChanged event, yet my event handler for the legacy winevent doesn't get called. So my app doesn't realize the caret has moved. So this means I need to ditch my use of the legacy winevent, and move to only use UIA for tracking the caret. This is probably something I would have done anyway at some point, but I now have a growing urgency. I wouldn't say this is my biggest regret in life, but I am kicking myself rather. I have little enough time to work on my app as it is, so to be adding to my workload simply because I chose to use legacy technology, really doesn't help. It's UIA-only for me now.

Using UIA to determine the bounding rectangle of the line of text where the caret is

So the goal with the new feature is to underline the line of text which currently contains the caret. That involves two steps. The first is to realize when the caret's moved, and the second is to get the bounding rectangle of the line of text where the caret is. My app already had code to react to the caret moving, and like I said above, that's not done by my app today in a way I'd recommend. Instead, I'd recommend that an app registers for notifications when the caret moves, by calling IUIAutomation::AddAutomationEventHandler(), passing in UIA_Text_TextSelectionChangedEventId. Your event handler will then get called as your customer moves the caret around the target app.

Note: Beware of what threads are being used in this sort of situation. Calling back into UIA from inside the event handler can cause unexpected delays, and I often avoid that by requesting when I register for the event, that certain data of interest relating to the element that raised the event, is to be cached with the event. Also, the WinForms app's UI typically won't be updatable from inside the event handler, so I may call BeginInvoke() off the app's main form in order to have the UI update made on the UI thread. Even having done that, I did once in a while find a COM interop exception thrown when trying to update the UI. I've not had a chance to figure out the cause of that yet, so I do have some exception handling in the code at the moment.

Ok, so let's say I know the caret's moved, and I need to find the bounding rect of the line of text which now contains the caret. I could take action every time the caret moves, to get the UIA element containing the caret, and check if it supports the UIA Text pattern. If it does support the pattern, get the bounding rectangle associated with the line of text containing the caret. But in practice, I don't need to check all that every time the caret moves. Rather, I could check whenever keyboard focus moves, does the newly focused element support the Text pattern? If it doesn't, then as the caret moves around that element, I know the Text pattern won't be available, and don't need to make the check for the pattern.

The code I ended up with is as follows:

// Cache the UIA element that currently has keyboard focus, so we don't need to retrieve

// it every time the caret moves.

private IUIAutomationElement _elementFocused;

// Call this in response to every change in keyboard focus or caret position.

public void HighlightCurrentTextLineAsAppropriate(bool focusChanged)

{

    // Are we currently highlighting the line containing the caret?

    if (!checkBoxHighlightTextLine.Checked)

    {

        // No, so make sure the window used for highlighting is invisible.

        _highlightForm._formTextLine.Visible = false;

        return;

    }

    // Using a managed wrapper around the Windows UIA API, (rather than the .NET UIA API),

    // I tend to hard-code UIA-related values picked up from UIAutomationClient.h.

    int propertyIdControlType = 30003; // UIA_ControlTypePropertyId

    int patternIdText = 10014; // UIA_TextPatternId

    // Are we here in response to a focus change?

    if (focusChanged)

    {

        // Hide the highlight until we know we can get the data we need.

        _highlightForm._formTextLine.Visible = false;

        // Create a cache request so that we access the data that we know we'll

        // need with the fewest number of cross-proc calls as possible.

        IUIAutomationCacheRequest cacheRequest = _automation.CreateCacheRequest();

        cacheRequest.AddProperty(propertyIdControlType);

        cacheRequest.AddPattern(patternIdText);

        // Until I figure out the occasional interop exception, wrap this in a try/catch.

        try

        {

            // Get the UIA element currently containing keyboard focus. (Note: If I rearrange some code

            // I could avoid making this call, and instead use the element which originally supplied the

            // UIA FocusChanged event.)

            IUIAutomationElement elementFocusNew = _automation.GetFocusedElementBuildCache(cacheRequest);

            if (elementFocusNew == null)

            {

                // If we failed to get the focused element, give up.

                return;

            }

            // For this first version, only work with Document and Edit controls.

            int CtrlType = elementFocusNew.CachedControlType;

            if ((CtrlType != 50030) && // Document

                (CtrlType != 50004)) //Edit

            {

                Debug.WriteLine("Newly focused element is neither Document nor Edit, so reset.");

                this._elementFocused = null;

            }

            else

            {

                Debug.WriteLine("Newly focused element is one of Document nor Edit, so use it.");

                this._elementFocused = elementFocusNew;

            }

        }

        catch (Exception ex)

        {

            Debug.WriteLine(ex.Message + " " + ex.StackTrace);

        }

    }

    try

    {

        // Do we know which element contains keyboard focus?

        if (this._elementFocused != null)

        {

            // Does the element support the Text pattern?

            IUIAutomationTextPattern textPattern = this._elementFocused.GetCachedPattern(patternIdText);

            if (textPattern != null)

            {

                // Get the current selection from the element. Even if there is no selection,

                // then we'll get data back relating to where the caret currently is.

                IUIAutomationTextRangeArray array = textPattern.GetSelection();

                if ((array != null) && (array.Length > 0))

                {

                    // For this version of the feature, only consider the first selection range

                    // if there are multiple selections in the app.

                    IUIAutomationTextRange range = array.GetElement(0);

                    if (range != null)

                    {

                        // Expand the range to encompass the entire line containing the caret.

                        range.ExpandToEnclosingUnit(TextUnit.TextUnit_Line);

                        // Now get the bounding rect for that line.

                        double[] rects = range.GetBoundingRectangles();

                        if ((rects != null) && (rects.Length > 3))

                        {

                            Rectangle rectLineText = new Rectangle(

                                (int)rects[0],

                                (int)rects[1],

                                (int)rects[2],

                                (int)rects[3]);

                            // We now know the bounding rect of interest. Move our highlight window

                            // so that it appears at the bottom edge of the bounding rect.

                            _highlightForm.HighlightCurrentTextLine(rectLineText);

                        }

                    }

                }

            }

        }

    }

    catch (Exception ex)

    {

        Debug.WriteLine(ex.Message + " " + ex.StackTrace);

    }

}

Perhaps one of the most exciting steps above is the call to ExpandToEnclosingUnit(). This is the place where I get to learn about the line that contains the caret. That function is really handy in other situations too. For example, if I want to learn of the bounding rectangle of a word or a paragraph, or of contiguous text which has the same formatting. That's pretty useful stuff in a variety of scenarios.

I should add that while the feature seems to hold up well enough in some apps, (including WordPad and NotePad,) it's not as reliable as I'd like it to be in other apps, (including Word 2016). I'll bet that's due to my use of the legacy technology I mentioned earlier. Ugh. I really need to find time to move to using only UIA, like I should have done in the first place.

Still, even with all the improvements I should look into, for a first version of the feature, it works well enough to generate the feedback that I need in order to make it as helpful as I can.

Figure 2: The line of text containing the caret being underlined in Word 2016.

Always keep in mind the accessibility of the app itself

Whenever I'm updating an app's UI, I need to consider the accessibility of the resulting UI. For this new feature, the only update to the UI is to add a radio button at a specific place relative to existing UI. Because I'm adding a standard control that's provided by the WinForms framework, I know I'll get a great head start on accessibility. For example, the control will be fully leverageable via the keyboard, and it'll be rendered using appropriate system colors when a high contrast theme is active, and the Narrator screen reader will be able to interact with the control. This is all great stuff, and in fact there were only two things I needed to check.

Focus order

As my customers tab through the UI, the order in which keyboard focus moves through the app must provide the most efficient means for my customers to complete their task. If keyboard focus were to bounce around the UI as the tab key is pressed, that would be at best a really irritating experience that no-one would want to have to deal with, or quite possibly make the app unusable in practice.

While my app's not web-based, the W3C web content accessibility guide Focus Order sums up the principal nicely for web UI, "focusable components receive focus in an order that preserves meaning and operability". I want that to be true in any app I build, be the UI HTML, WinForms, WPF, UWP XAML or Win32.

Fortunately, Visual Studio makes it quick 'n' easy for me to make sure I'm delivering an intuitive tab order. All I do in Visual Studio is go to View, Tab Order, and then select each control in the UI, (using either the mouse or keyboard,) in the order I'd like keyboard focus to move through the UI.

The screenshot below shows all the controls in the app UI with an accompanying number shown by each control, indicating the control's position in the tab order.

Figure 3: The app form in Visual Studio's design mode, with the tab order shown by the controls in the form.

Note that when using the Tab Order feature, it is important to include the static Labels in the logical place in the order, even though keyboard focus doesn't move to the Labels as your customer tabs around. For some types of control, if an accessible name has not been set on the control, then WinForms might try to leverage an accessible name based on the text of a nearby Label. For example, with a TextBox or ComboBox which don't have a static text label built into the control. In those cases, having the associated Label precede the control in the tab order, can result in the control getting the helpful accessible name that your customer needs.

Programmatic Order

Whenever I'm updating UI, I need to consider both the visual representation of the app, and the programmatic representation as exposed through UIA. Both of these representations must be high quality for my customers.

In some situations, the path my customers take when navigating through the UI will be based on the order in which the controls are exposed through the Control view of the UIA tree. The Control view is a view which contains all the controls of interest to my customers, including all interactable controls and static Labels conveying useful information. (So the view might not contain such things as controls used only for graphical layout which are not required to be exposed either visually or programmatically.)

Having added the new CheckBox to the app, I pointed the Inspect SDK tool at the UI to learn where the control was being exposed in the UIA hierarchy. It turned out that the CheckBox was being exposed through UIA as the first child element beneath the app window. So programmatically, it existed before all other elements in the UI. The screenshot below shows the CheckBox is being exposed before all the other elements, which are its siblings in the UIA tree.

Figure 4: Inspect reporting the UIA tree of the app, with the new CheckBox as the first child of the app window.

So say a customer using the Narrator screen reader encounters the app window. If they're not familiar with the app, they might choose to switch to use Narrator's Scan mode, to learn about the UI. By doing that, they may press the up and down arrows and move through the controls in the UI, including the controls which can't get keyboard focus. And the navigation path then taken through the controls must be a logical one based on the functionality in the app. Importantly, the path actually taken is impacted by the order of the elements as exposed through UIA. This means the first element they encounter will be the new CheckBox. Then they'll move to the static Label shown visually at the top of the app. And later, they'll move from the control shown visually before the new CheckBox, directly to a control following the new CheckBox. This is not the experience I want to deliver at all.

So to address this, I edit the designer.cs file for the app, and change the order in which the controls are added to the form. After I originally added the new CheckBox to the app, the related designer.cs code was as below. I've highlighted the line of interest in the code, which contains the new control called "checkBoxHighlightTextLine".

//

// HerbiHocusFocusForm

//

resources.ApplyResources(this, "$this");

this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;

this.Controls.Add(this.checkBoxHighlightTextLine);

this.Controls.Add(this.labelInstructions);

this.Controls.Add(this.checkBoxHighlightFocus);

this.Controls.Add(this.checkBoxCaretAbove);

this.Controls.Add(this.checkBoxCaretBelow);

this.Controls.Add(this.groupBoxCustomise);

So I grabbed the line adding the new CheckBox to the form, and I moved it to be between the lines which add the controls logically before and after the CheckBox. I've highlighted the line of interest in the following resulting code.

//

// HerbiHocusFocusForm

//

resources.ApplyResources(this, "$this");

this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;

this.Controls.Add(this.labelInstructions);

this.Controls.Add(this.checkBoxHighlightFocus);

this.Controls.Add(this.checkBoxCaretAbove);

this.Controls.Add(this.checkBoxCaretBelow);

this.Controls.Add(this.checkBoxHighlightTextLine);

this.Controls.Add(this.groupBoxCustomise);

Having done that, if I now point Inspect at the UI, it reports that the CheckBox element is sandwiched between the other controls in the UIA tree in a manner that matches the meaning of the UI.

Figure 5: Inspect reporting the UIA tree of the app, with the new CheckBox exposed in the appropriate place in the UIA tree.

So what's next?

Over the next few weeks I hope to grab a few early hours here and there, and work on some of the points raised by people using the app. These include:

1. Customizable color, size and transparency for the underline. That should be quick to do, so I'll probably work on that first. And as always, I'll make sure the app's tab order and UIA hierarchy are intuitive after updating the UI.

2. I got the feedback, "her last wish is that the focus marking would work even better and would work in the Windows menu etc." This is going to be an interesting one, and I'll need to follow up to learn more about exactly which UI is of interest. For regular app menus, my focus highlight struggles, because the menu can appear on top of my focus highlight. I expect I can address that, by adding an event handler such that I can learn when my highlight isn't top-most, and then moving my highlight back on top when necessary. (And I'd need to make sure I can't get stuck in a loop with other UI which also tries to keep itself on top.) But what I suspect is really the request here, is that the highlight works on the Start menu, and that's not straightforward for me to resolve. For my app to achieve that, it would need UIAccess privileges, and as I understand things, apps downloaded from the Store today can't get that. So the only way I could get that to work would be for me to revert to shipping my app through my own site and installer, and signing it, which I'm really not set up for at the moment. Hmm. I'm not sure what I'll do about this.

3. The app needs to work more reliably in some apps, and in some cases, work at all. One app which I'm particularly interested in is Edge, when caret browsing's turned on. For this to happen, I need to ditch the code I have relating to use of the legacy winevent, and move to only use UIA.

4. This last point is not something I've had feedback on, but is something I'm curious about nonetheless. Sometimes my underline appears further below the line of text than might seem appropriate, and I expect that's due to the paragraph formatting on the line. So I wonder if I can get what Word calls the "Spacing after" for the line of text, and account for that when positioning my underline. Perhaps this would keep the underline at the same distance from the text shown visually on the line, regardless of the spacing after. I really don't know if that's possible, but it'll be interesting to find out.

Summary

It's been a pleasure for me to discuss with the organization which asked for the new underline feature, the human aspects of building software like Herbi HocusFocus. For me, software has always been a means to an end. That end being the impact it has on people's lives. Other feedback I've received from the organization is:

"I also agree with the fact the developers should look at the way they can create tools for people in their community. They can see it as a form of charity work, giving back to their community and they'll see how small tools or adjustments can mean a world of difference to people. Software developers should always wonder how their users will actually use their software and build it with their users in mind. People and usability come before the technology if you ask me. That's why accessibility and a user friendly design should be getting more attention. All users benefit from a good design and from accessibility options and some users even depend on them to be able to use a piece of software. I'm also a strong advocate for user testing before launching a product, most software developers have certain expectations of how their users will interact with their software. Sometimes they can be quite wrong about the way people use and view their software and which steps seem logical to the user."

I can't argue with any of that. In fact, after more than sixteen years of building exploratory assistive technology apps like Herbi HocusFocus, only a handful of my apps actually had any impact. And those were the apps where someone had contacted me, specifically asking whether I could build a tool for them, because there didn't seem to be a solution available to them already. I'm very grateful to have been able to learn from all those people, and to get a better understanding of where I can have most impact.

Overall, this exercise has been a reminder for me of how UIA can help devs add some seriously useful functionality to an app, with relatively little work. That doesn't mean to say UIA does everything you want. I once told a dev how UIA doesn't provide a simple way to get a collection of elements that lies within a rectangle. He replied saying "I find that difficult to believe". Well, we live in the world we live in, and there's lots of things that we might like to exist in this world, and they don't. As far as I'm concerned, we try to improve things for the future, and make the most of what's available to us today. And I believe that UIA has a lot to offer us and our customers today.

So please do consider how UIA might help you provide a powerful new feature to help your customers. Even with all my new feature's constraints, the first piece of feedback I got, was "Wow, I'm amazed by your actions!". Anyone who knows me, knows my actions are far from amazing. But I have the support of some very cool technology that can make me look pretty useful at times.

Guy

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Introduction

When considering how an app might be used with different methods of input or output, it's not enough to simply consider whether it can be used at all. It's also very important to consider what's the most efficient way in which the app can be used. With that in mind, some colleagues and I have built a simple solitaire app, aimed at generating discussion on efficient ways in which all input and output methods available on Windows might be used to play a game of solitaire. As the app is today, it's very much a "V1", with lots of opportunities to add more functionality. But hopefully it is at a point where people can get back to us and say: "Yes, I get how technically the app can be used in a bunch of ways, but what would work great for me, is …".

While the app explores many types of input and output methods, it's by no means a comprehensive set. In fact I wouldn't say that I know what a "comprehensive set" of input and output methods is. But to get the ball rolling on the discussion, the following input and output methods have been explored with the app to one degree or another.

• Touch
• Mouse
• Pen
• Keyboard via tabbing and arrowing, and use of the Space and Enter keys
• Keyboard via access keys
• Windows Speech Recognition
• Eye Control
• Switch device control
• Default visuals
• Dark app mode
• Windows High Contrast themes
• Windows Display Scaling
• Windows Magnifier
• Narrator screen reader

A video briefly showing all the input and output methods listed above, is at The Sa11ytaire app for Windows 10. And yes, it's the typical sort of low quality video that I always make. (At least my dog stayed quiet throughout the filming.) And I've yet to add captions to the video.

In later versions of the app we can concentrate on other topics not discussed in this post. For example, configuring the size of text shown in the app, independently of the other visuals shown. And considering how announcements made by the app which include multiple lines of text, relate to the braille device experience. And how some success/failure notifications which are audio-only today, can have some matching visual output. Basically – there's no end of interesting topics to consider in an app like Sa11ytaire.

Another important aspect of the app is to consider how different types of input and output methods can be used together. It's not sufficient for the app to be usable with one of the methods listed above, but only so long as one of the other methods isn't also being used. For example, you use a switch device, and also prefer visuals shown using colors from the High Contrast Black theme that comes with Windows. In that case, the highlighting that cycles through the app as the switch device is used, had better be shown with an appropriate color from the High Contrast Black theme. Or say you use Windows Speech Recognition (WSR) to input at the app, and have the Narrator screen reader announce details about the state of the app. That's a particularly interesting case, given that both WSR and Narrator will leverage the programmatic representation of the app exposed through the UI Automation (UIA) API. So what single UIA representation can provide the most efficient experience when WSR and Narrator are interacting with the app at the same time?

Now, that all said, the current V1 of the Sa11ytaire app does not support all combinations of the various input and output methods listed above. But surely there's no reason why a later version of the app couldn't.

Figure 1: The default visuals of the Sa11ytaire app.

Key Learnings

The Sa11ytaire experiment has been a fascinating project to work on, and I've learnt a number of things in the process. But as I think back on this, there are two things that really stick in my mind.

Use standard controls that come with the UI framework

Many of the UI frameworks that you may be using on Windows, (for example, UWP XAML, Edge/HTML, WPF, WinForms, Win32), will do a ton of work on your behalf to make your app as accessible as possible by default. It often won't be able to make the app fully accessible by default, but by using standard controls that come with the framework, you'll get a great head start. The interactable elements presented in the Sa11ytaire app may not all visually appear to be standard controls, but they are all based on standard controls. In fact every interactable element in the main area of the app, is based on one of the following standard controls: Button, ToggleButton, ListView or ListViewItem. As such, the elements are going to support input methods such as touch, keyboard and eye control by default. And they'll all support being programmatically controlled through the UIA API, (which is essential for some types of input such as touch control of Narrator), again, without me having to take any specific action to enable that. By default, I would always base an interactable element on a standard control, regardless of whether it shows custom visuals.

If I pushed a UI change without testing both the programmatic and visual representation, then I pushed it without testing it

Typically when I'm working on a project like this, things are pretty frantic. I'll try to squeeze in a fix or update in the time I have, and then get on with whatever I was meant to be doing. During the app's development, I made a change to the UI relating to the cards being moved to what we called the "target piles". I ran the app, and the change looked good, and I pushed it to the repo. Before long, Tim, a dev collaborator on the app, pulled the change and let me know that after he'd moved a card to a target pile, his screen reader said that the target pile was still empty. So I'd done exactly what I ask devs not to do. I'd verified the visual representation of the app after my UI change, but not the programmatic representation. If I'd taken a few seconds to point the Inspect tool that comes with the Windows SDK, at the UI I'd changed, I would have learned I'd broken the app. So no matter how rushed I feel, if I don't verify both the visual and programmatic representation of the app, then I'm pushing a change without testing it. If I'm working on an app in collaboration with others, I have a responsibility to do better than that.

Methods of input and output

The sections below detail some of the input and output methods used with the Sa11ytaire app.

Touch

Touch control of the app worked by default, due to the use of standard controls in the app.

Mouse

Mouse control of the app worked by default, due to the use of standard controls in the app.

Pen

Now, someone might ask here, why would anyone want to play solitaire with a pen? But at this point I think a more interesting question is, if someone wants to play solitaire with a pen, can they?

Pen control of the app worked by default, both directly at the screen and with an external tablet, due to the use of standard controls in the app.

Keyboard via tabbing and arrowing, and use of the Space and Enter keys

Keyboard control of the app worked by default, due to the use of standard controls in the app.

This meant that the Tab key can be used to move keyboard focus between the Button, ToggleButtons and ListViews in the app, and Arrow keys could move between the ListViewItems in the ListViews. A press of the Enter or Space keys would then perform the appropriate action associated with the focused control. For example, the "Next card" element would be invoked, the "Upturned card" or "Target pile" elements would be toggled, and a "Dealt card" element would be selected.

Given that it's not efficient to have to tab through every interactable element in the app in order to reach a specific element of interest, we added support for a press of the F6 key to move keyboard focus to the first focusable element in each of the three main areas of the "Remaining cards", the "Target piles", and the "Dealt card piles".

Now, we did have an opportunity to start experimenting here. Often in the lists of dealt cards, some of the cards are face-up and some are face-down. Given that there's no action you can trigger with the face-down cards, we took action to disable them. This means keyboard focus can't reach them. On the one hand, this seems appropriate for the requirements of the game. But on the other hand, this isn't how lists typically work. You rarely encounter lists with a mix of enabled and disabled items. And one of the essential ingredients in the recipe for a great experience, is "intuitive". A colleague who uses a screen reader pointed out that it could be confusing to be told that you're at (say) the fourth card in a list of four cards, but then you can't arrow to any other card in the list. If this model really is going to be used going forward, then we'll need to update the app such that it's clearer to the person playing the game as to exactly what's going on. So for a later release we'll do that, unless we decide that overall it's preferable to have all the cards enabled, and take action to prevent a face-down card from being selectable.

Regarding one interesting implementation detail, say a list of cards has multiple face-down cards and one face-up card. Originally when we moved the face-up card away from that list, we enabled what was previously the face-down card beneath the card being moved. Sometimes we'd find that at that point, the newly enabled card was unexpectedly not in the tab order in the app. No amount of re-ordering the action taken in the code seemed to prevent that issue, and the enabled card would remain out of the tab order until it was selected somehow. So in order to account for that, the code's action is rather unusual now in that it always maintains at least one enabled item in the list. That item might change its appearance as cards are moved around, but it always stays in the list. Even when the list is "empty", it still contains an item which is the empty slot where a king can be placed.

Another aspect of keyboard usage which we'd like to explore more, is where would it be most helpful for keyboard focus to end up after specific actions? For example, say you tab over to a target pile and press Enter to move a selected card in a dealt card pile over to the target pile. Today, after taking that action, keyboard focus is left at the target pile. But might it be more helpful for keyboard focus to be set back in the dealt card pile from which the card was just moved? After all, it's probably more likely that you'll want to interact with that dealt card pile than the target pile. We actually tried doing that, but found that when we programmatically moved keyboard focus back to the dealt card pile, if Narrator was running, Narrator didn't follow that change in keyboard focus. Instead Narrator was left at the target pile, and so we decided to stick with the original design of having keyboard focus left at the target pile. We'll revisit this later if moving keyboard focus to the dealt card pile does seem like the most helpful thing to do in this situation.

Keyboard via access keys

While it's essential to provide an intuitive keyboard experience like that described above, some people might find an alternative method of using the keyboard more efficient. And that method is: access keys! With the Windows 10 Fall Creators update, it's absolutely trivial for a dev to add access keys to an app like Sa11ytaire, and so provide a very efficient experience for people who leverage those access keys.

To add an access key involving a single character to a control, add AccessKey="<some character>" in the XAML. For the Button and ToggleButtons in the Sa11ytaire app, use of the access keys results in those controls' Click and Checked event handlers being called. For each of the ListViews containing the dealt cards, we added the AccessKey value and also an AccessKeyInvoked event handler. When that handler was called, we took action to select the last item in the associated list. (For lists showing multiple face-up cards, this might be followed by presses of the Up Arrow key to reach another card in the list.)

With those few changes, a press and release of the Alt key will show all the available access keys. I think the access key experience is really slick, given that the Alt key can be kept depressed throughout the action to move a card. So to move a card from dealt pile 2 to dealt pile 6, press Alt+2+6. Or to move the upturned card to the Hearts target pile, press Alt+U+H.

I've said it before, and I'll say it again: Few things in life have a better dev effort-to-customer benefit ratio than access keys.

Figure 2: Access keys shown in the app in response to a press of the Alt key.

Windows Speech Recognition

In order to deliver a great experience with Windows Speech Recognition (WSR), it's important to know the programmatic representation of the app, as exposed through the UI Automation (UIA) API. Many standard controls that come with the UWP XAML framework are programmatically accessible by default. For example, say you add a Button control, set its content to some localized string like "Save", and add a Click handler. A UIA client app like the Narrator screen reader will say "Save, button" when it encounters the Button, and can programmatically invoke the Button through the UIA API. All that programmatic accessibility is provided by default through the use of a standard control.

In the Sa11ytaire app, the UI on the interactable elements has been customized to present specific visuals. For example, the dealt cards show a suit symbol in addition to a text string which specifies the value and suit of the cards. And an empty target pile shows nothing but the relevant suit symbol. So unless I take action to examine the programmatic representation of the app, I can't really be sure how my customization might have affected that programmatic representation. So in order to learn of the programmatic representation, I'll point the Windows SDK Inspect tool at the app.

Note: While I'd never assume this, it might be possible that I could technically play the game with WSR, without any specific action on my part around the programmatic representation of the app. For example, I expect I could use the WSR "Show Numbers" feature to have a unique number shown over every interactable element, and then say that number to interact with the element. But that's not at all the experience that I want people to have to rely on. For example, if I want to select the 2 of Hearts card, I don't want to have to say something like "Show numbers, 12". Rather I want to say "2 of hearts", because that provides an intuitive and efficient experience. And "efficiency" is another essential ingredient in the recipe for a great experience.

The UIA Name exposed by the various UI elements is managed in different ways in the app depending on the element. The "Next card" Button-based element takes no specific action at all. For that element, the UWP XAML framework simply leverages the element's Content property as the UIA Name. So whether it's visually showing "Next card" or "Turn over cards", that string becomes the UIA Name property.

Note: The current version of the app is not localized. If the visual strings are localized at some point, then action would be taken to verify that all the UIA Name properties of the elements in the app are also localized. A UIA Name property must be accurate, concise, unambiguous, and by default localized. (I say "by default", given that such things as brand names would often not be localized in either the visual or programmatic representation.)

For the ToggleButton-based elements, there are calls to AutomationProperties.SetName() in code-behind to have the appropriate UIA Name exposed for those elements. While this works fine, at some point I expect we'll update the app to have the AutomationProperties.Name for those elements bound in the XAML to some property on the element.

For the dealt cards in the ListViews, the AutomationProperties.Name is bound to a custom property on the outermost Grid in the DataTemplate defining the card's UI. That property can return three forms for the UIA Name, depending on the situation. These forms are either the visible text string, (such as "2 of Hearts",) or a string containing "Empty", (when all cards have been moved out of the list,) or a string containing "Face-down", (because a face-down card's value is not to be exposed either visually or programmatically). In the case of the "Face-down" values, they have the list index appended, to ensure that any such UIA Name is unique relative to its siblings.

The dealt card's XAML for that bound Name property is as follows:

AutomationProperties.Name="{x:Bind Name, Mode=OneWay}"

And the code-behind for the associated custom class is below. This class is set to be the x:DataType in the DataTemplate which is set as the ItemTemplate for each dealt card pile ListView.

public string Name

{

    get

    {

        string name;

        if (this.FaceDown)

        {

            name = "Face-down " + this.InitialIndex;

        }

        else if (this.card.Rank != 0)

        {

            name = this.card.ToString();

        }

        else

        {

            name = "Empty " + this.ListIndex;

        }

        return name;

    }

}

Having done the above, the Inspect SDK tool shows me that all the interactable elements in the app have appropriate UIA Name properties.

Figure 3: The Inspect SDK tool reporting the UIA Name properties on the elements in the Sa11ytaire app.

And with those UIA Names verified, I can then use Windows Speech Recognition to efficiently play the game.

Figure 4: Using WSR to play the Sa11ytaire app, with the WSR UI showing "4 of Diamonds" after the 3 of Clubs card has been moved to the 4 of Diamonds card.

By the way, the screenshot of the Inspect SDK tool shown above, shows that each card in the dealt card list is exposed through the Control view of the UIA tree as a single UIA list item element. (The Control view should contain only elements of interest to the person playing the game.) So the next question is, given that the outermost Grid in the card's DataTemplate contains multiple TextBlocks, how come those TextBlocks aren't also exposed through that UIA Control view? The answer is that those TextBlocks have the following set on them in XAML:

AutomationProperties.AccessibilityView="Raw"

By setting an AccessibilityView of "Raw", we're saying that the element should only be exposed through the Raw view of the UIA tree, and not the Control view. That means that while the element is still being exposed through UIA, it's being done in such a way that screen readers are being told that the element is not of interest to the person playing the game. It's reasonable to do that in this case, because all the information that the person playing the game needs is accessible through the containing list item element. To be able to navigate with the screen reader to the contained TextBlocks probably wouldn't be serving any useful purpose.

Eye Control

The Windows 10 Fall Creators update shipped with built-in support for eye control, so this is perfect timing for the Sa11ytaire app. I'd not used the eye control before, so I borrowed an eye tracking device and tried it out. To my delight, the Sa11ytaire app was useable via eye control by default, without any changes being required at all to the app.

Once again, use of standard controls in the app had led to the app being accessible by default for customers using a particular input method.

Figure 5: The Windows Eye Control UI showing over the Sa11ytaire app, and being used to move a 10 of Clubs over to a Jack of Hearts.

I have to say, my classic feeling of being rushed at all times did trip me up when first trying out eye control. When the eye control UI appeared, I'd never seen it before, and I went straight to the left mouse button icon at the top left of the UI, given that that was the equivalent of where I'd go to click the left mouse button on a physical mouse. Having done that, I could interact as expected (or so I thought) with the "Next card" button, and the cards in the dealt card lists. I could not, however, interact as expected with a card in the upturned cards pile. It turned out that I was using eye control to trigger a left mouse button double-click rather than a single-click. While a double-click led to some action that I thought was expected in some places in the app, it left elements based on ToggleButtons in their original state. What I should have been doing is going to the bottom left corner of the eye control UI to trigger a left mouse button single-click. Once I did that, everything worked as expected. So I'd recommend that you spend a little more time than I did with the eye control feature before wondering if all is not well with ToggleButtons.

Switch device control

The bulk of the input action taken in the Sa11ytaire app is to first select a "source" element, and then select a "destination" element. And that's it. And so it should be relatively straightforward to update the app to have it usable with a single-switch device as the input method. A press of the switch would start a highlight moving through all the interactable elements, and when it reaches an element of interest, another switch press would trigger the appropriate action at the element. That action might be an invoke, toggle or select, depending on the element. Repeating that process with a different element would complete the action required to move a source element to a destination element.

So the Sa11ytaire app was updated to have limited support for a single-switch device. A press of F10 will toggle switch device control mode. The app assumes today that a press of the switch device leads to the app receiving a Space key press, but can be updated to handle different types of input in the future based on feedback. The current implementation means that switch device control can be simulated without a switch device, simply by pressing the Space key.

The app does support different speeds of highlight while in switch control mode, and that's configurable through the AppBar.

One approach that we could take here is to create a timer, and on every timer tick, move keyboard focus to the next interactable element in the app. That would be relatively straightforward to do with the FocusManager. Then, once keyboard focus is on the element of interest, have the next switch press trigger some action at the element with keyboard focus. While that would technically work, again we wanted to consider how we could make the experience more efficient. So we built the feature such that the highlight first moves between the three main areas of the remaining cards, the target piles, and then the dealt card lists. Once the highlight is on an area of interest, a press of the switch device starts highlighting the elements within that area. And then depending on the area, a subsequent switch device press will trigger action on an element. I say "depending on the area", because a dealt card list with multiple face-up cards also needs to scan through the face-up cards, before having a switch device press select one of those cards.

As part of providing this more efficient experience of highlighting the three main areas, we needed to consider options for how that might be implemented. If those areas were implemented as keyboard focusable elements, then we could just move keyboard focus to them. But they intentionally weren't keyboard focusable, given that they're not interactable when using the keyboard as the input device. So instead, when we want the switch device control highlight to show at one of those areas, we take action to update a property of the related element resulting in a visual indication of what will react to a switch device press at that time. Depending on the element, the properties updated are either the BorderBrush and BorderThickness, or Background.

Important: With this change, we've now entered the world of seriously providing custom visuals, and that means we need to consider how this relates to everyone who uses the app. For example, if a high contrast theme is active, the custom highlight visuals had better use an appropriate color from that active theme. And how will the Narrator screen reader let people who are blind know what's going to react to a switch device press? We'll discuss those topics later.

It's recognized that the switch device control feature in the app is far from complete. The switch device control highlight does not move to the AppBar today, and so we added a "Restart game" button which is shown when switch device control is on, and that button is included in the highlight path through the app. In order to reduce the chances of the app being restarted unintentionally, a confirmation dialog appears when that Restart game button is invoked.

Note: Any dialog that appears in the app must be controllable through a switch device, when switch device control is on. In order to achieve this, a timer is started when the dialog appears, and on each timer tick, keyboard focus moves to the "next" control in the dialog. A press of the Space key invokes the button with keyboard focus by default, because when the dialog's up, the app's key event handler isn't called. The first attempt at implementing this had the timer tick handler calling the FocusManager's TryMoveFocus(), and that seemed to work ok. But I then noticed that sometimes the visual feedback for keyboard focus disappeared while the dialog was up. I suspect keyboard focus was still moving, but the visual feedback had gone. Perhaps this is related to how when keyboard focus is programmatically moved, visual feedback for the focus may or may not appear depending on the situation. So I replaced the use of TryMoveFocus(), with a call to FocusManager.FindNextFocusableElement() to get the "next" button on the dialog, and then a call to Focus() off that button, specifically passing in FocusState.Keyboard. With that change, I've yet to find the visual feedback for keyboard focus disappear while keyboard focus moves between the dialogs' buttons.

So while the switch device control feature is in its early days at the moment, I am hoping it's sufficient to generate feedback on how the switch device control can be made more efficient for this app.

Figure 6: Switch device control highlighting moving through the face-up cards in the last dealt card list.

And here's a note on another interesting point that cropped up during the development of the switch device control feature. Originally, I'd used the term "scan" to refer to the switch-related functionality, as that seemed fair enough for the highlight as it cycles through the various parts of the app's UI. But then I was asked, if switch device control was turned on and the app said "Scan mode on", what would that mean to someone who's also using Narrator, which has its own feature called "Scan mode"? This was a great point. While ultimately I want as many input and output methods as possible to be usable concurrently, there mustn't be any ambiguity around the terminology. So when referring to the switch device control feature in the app's UI, I changed the terminology from "scan" to "switch control". A work item still outstanding for me is to replace all the uses of "scan" in the source code too. There must be no ambiguity for people playing the game, or for devs working in the code.

Default visuals

The default visuals for the Sa11ytaire app are intended to present something similar to a typical experience for a card game. So the face-up cards have text and symbols shown in either black or red on a white background, and everything's on top of a green app background. Today the app visuals are not sophisticated at all. The visuals are functional, but not elegant, and that's fine for the app at this stage. The intention at the moment is to explore various input and output methods, not to spend a lot of time on delivering more attractive visuals.

Important: While it is recognized that the app's visuals are not elegant, this is in no way meant to imply that an accessible app doesn't need beautiful visuals. A great app delivers a delightful experience for all people, including those who can consume beautiful visuals. We mustn't build an app with an attitude that the app will be either accessible or visually beautiful, but not both. The rather utilitarian visuals presented by the app today are just another example of where the app has certain constraints, but not so limiting as to prevent people from providing feedback.

There are two very important considerations around the default visuals, and which apply to any app presenting UI.

1. No important information should be conveyed through color alone.

The colors shown in the app enable some people to determine at glance whether card is a club/spade, or a diamond/heart, and that can help them more efficiently play the game. But for other people who don't leverage the colors shown on the card, the game can still be played. The symbol used to represent a card's suit, and the text shown on the card, convey a superset of the information conveyed through the use of color.

When considering the use of color in an app, it can be interesting to go to the "Color filters" setting in the Settings app in the Windows 10 Fall Creators update. By examining an app's UI while different color filters are active, you can consider whether the efficient usability of the app has been impacted by the filter.

Figure 7: The Sa11ytaire app presented when the Grayscale color filter is active.

2. Contrast of text against its background.

When I encounter text in an app, I want the text to jump out at me. I don't want to have to lean in and take time to figure out what the text says. Light blue text on a slightly darker blue background might look cool to some people, but not to me. The more time I have to spend trying to consume important text in an app, the less I want to use that app.

So when creating the visuals, I ran a color contrast analyzer tool to check the contrast of the text shown in the app, against its background. As far as I was concerned, if the text has a contrast of 4.5:1 against its background, (which is the minimum default contrast required for business apps that I work with,) then I'd consider that to be sufficient for the Sa11ytaire app. While I knew black text on a white background would be fine, what about the black text on a blue, green or gray background, or red text on a white background? As it happened, the color contrast analyzer reported that in the app's default state, all the text met the minimum color contrast requirement.

However, things are not ok! I've verified the color contrast for the text on elements in their default state, but what about other states? For example, what happens if the background behind text gets darker on mouse hover? Sure enough, on mouse hover over some elements in the app, the background of the elements darkens, the color contrast of the text against the background becomes 2.1:1, and the text is almost illegible. Someone might say, "So what? You knew what the element said before you moved the mouse there". But really, that's not a delightful experience. A great app shows easy-to-read text regardless of the state of the element showing the text. And on a similar note, in the earlier Figure 6 screenshot, some switch device control highlight shows a gold border against white, with a color contrast of 1.4:1. That highlight might not be text, but in this case, the highlight used for switch device control should always stand out. So all in all, there are definitely some fixes required here, and I just haven't made them yet.

On another note, it's also worth considering what it would mean if important information was only conveyed through the use of a shape in the app. When the app starts, the four target piles show only the symbol for the suit associated with the target pile. If someone using the app found it a challenge to remember what each symbol represented, or found it a challenge to differentiate between the symbols, how would that impact the usability of the app? That's something I'd like to explore further, but this version of the app doesn't take any specific action to account for that.

Dark app mode

While discussing the Sa11ytaire app, a colleague who has low vision pointed out that she would prefer to use the app with the dark app mode Windows setting active. I've never created an app with colors that are specific to dark app mode, so I updated the app.xaml file to add a specific set of colors in its "Dark" ResourceDictionary.

This app feature is not so much "far from complete", as it is "not really started". I threw some colors in, but have no experience around what colors might typically be most helpful here. (And in the screenshot below, some black text has extremely low contrast against its dark green background.) So I'm mentioning support for the dark app mode here, not because the app has something useful to show today, but because I consider it absolutely necessary for the app to provide useful support for the dark app mode at some point. I'm looking forward to discussing this more with my colleague, and if you know of a set of dark app mode colors that would work well for you, do let us know.

Figure 8: The Sa11ytaire app showing non-default colors when the Windows dark app mode setting is on.

Windows High Contrast themes

I mentioned above that I'd updated the app to contain "Dark" theme colors, in addition to the "Default" (or "Light" in this app's case) theme. My next step was to update the "HighContrast" ResourceDictionary in the app.xaml file to have the app show only appropriate system colors when a high contrast theme is active.

The following is the set of system colors chosen for the various elements in the app UI.

<ResourceDictionary x:Key="HighContrast">

    <SolidColorBrush x:Key="CardTableBackgroundBrush" Color="{ThemeResource SystemColorWindowColor}" />

    <SolidColorBrush x:Key="NextCardBackgroundBrush" Color="{ThemeResource SystemColorButtonFaceColor}" />

    <SolidColorBrush x:Key="NextCardEmptyBackgroundBrush" Color="{ThemeResource SystemColorButtonFaceColor}" />

    <SolidColorBrush x:Key="NextCardForegroundBrush" Color="{ThemeResource SystemColorButtonTextColor}" />

    <SolidColorBrush x:Key="NextCardEmptyForegroundBrush" Color="{ThemeResource SystemColorButtonTextColor}" />

    <SolidColorBrush x:Key="TargetPileCardBackgroundBrush" Color="{ThemeResource SystemColorButtonFaceColor}" />

    <SolidColorBrush x:Key="TargetPileEmptyBackgroundBrush" Color="{ThemeResource SystemColorButtonFaceColor}" />

    <SolidColorBrush x:Key="PlayingCardBackFaceUpBackgroundBrush" Color="{ThemeResource SystemColorButtonFaceColor}" />

    <SolidColorBrush x:Key="PlayingCardBackFaceDownBackgroundBrush" Color="{ThemeResource SystemColorButtonFaceColor}" />

    <SolidColorBrush x:Key="PlayingCardBackPlaceHolderBackgroundBrush" Color="{ThemeResource SystemColorButtonFaceColor}" />

    <SolidColorBrush x:Key="SuitBlackForegroundBrush" Color="{ThemeResource SystemColorButtonTextColor}" />

    <SolidColorBrush x:Key="SuitRedForegroundBrush" Color="{ThemeResource SystemColorButtonTextColor}" />

    <SolidColorBrush x:Key="CardScannedBorderBrush" Color="{ThemeResource SystemColorHighlightColor}" />

    <SolidColorBrush x:Key="CardBorderBrush" Color="{ThemeResource SystemColorButtonTextColor}" />

</ResourceDictionary>

Most of the above system colors are the colors associated with the text and background of buttons. Note that I don't care what those colors actually are. They could be white on black, yellow on black, black on white, or some other colors specified by the person playing the game. All I care about is that the app presents the colors that the person using the app expects.

Given that I don't know what colors will end up being shown in the app, I mustn't hard-code any color in the app. Accounting for that can sometimes be a challenge when you're designing an app, given that you may have many things you'd like to show using different colors, and only a relatively small set of system colors to choose from when a high contrast theme is active. In the case of Sa11ytaire, the choice of colors was pretty straightforward. For example, the color of the switch device control highlight is the SystemColorHighlightColor.

An interesting aspect of supporting system colors in the app relates to the fact that many elements might show colors specific to their current state. For example, the background of a card in a dealt card list is affected by whether the card's face-up or face-down, or in fact represents the empty slot where a king can be placed. In order to account for that, a number of ValueConverters are used when binding visual properties to custom properties on the related classes. The example below is the converter for converting the card's "State" custom property to the UI element's Background property. I don't know if that's the recommended way to do all this, but it seemed to work fine, and accounted for whether a high contrast theme is active or not.

public class IsCardStateToCardBackgroundConverter : IValueConverter

{

    public object Convert(object value, Type targetType, object parameter, string language)

    {

        CardState state = (CardState)value;

        SolidColorBrush backgroundBrush;

        switch (state)

        {

            case CardState.FaceDown:

                backgroundBrush = Application.Current.Resources["PlayingCardBackFaceDownBackgroundBrush"] as SolidColorBrush;

                break;

            case CardState.KingPlaceHolder:

                backgroundBrush = Application.Current.Resources["PlayingCardBackPlaceHolderBackgroundBrush"] as SolidColorBrush;

                break;

            default:

                backgroundBrush = Application.Current.Resources["PlayingCardBackFaceUpBackgroundBrush"] as SolidColorBrush;

                break;

        }

        return backgroundBrush;

    }

    public object ConvertBack(object value, Type targetType, object parameter, string language)

    {

        throw new NotImplementedException();

    }

}

One thing that I'd like to do but not yet found a trivial way of doing, is to have the entire app's UI updated in response to a change in state of high contrast while the app's running. It's easy to add an event handler to react to a change in the state of high contrast, but in response to that, I want to take some simple "Refresh all app UI to account for the new state of high contrast" action. As things stand today, the app has to be restarted after a change in state of high contrast. That might not seem like a big deal, but say someone using the app finds it a challenge periodically to use the default visuals while the app's running, and wants to temporarily move to use high contrast colors in order to continue their current game. So they issue the Windows keyboard shortcut for moving into high contrast, and expect the app to react accordingly. Well, not today it won't.

Note: The app does make use of an interesting UWP XAML "HighContrastAdjustment" feature which I've not used before. Traditionally, many apps have defined cool default visuals, and unfortunately not considered the experience for people who use high contrast themes. That meant that in the past, the default colors could continue to be shown when a high contrast theme is active, and the app becomes unusable. Because of this, the UWP XAML framework now tries to reduce the chances of this sort of broken experience from reaching people using the app. UWP XAML can now automatically render elements using colors based on the active high contrast theme, even if the app wasn't built to explicitly request that. For example, static text will be rendered using the system color for static text, and text on buttons will be rendered using the system color for text on interactable controls. While this can significantly improve the experience for people using high contrast themes, the Sa11ytaire app fully manages the colors to be used when a high contrast theme is active. And the app has a better understanding of exactly which color should be used in cases such as a ListViewItem containing a TextBlock. As such, the app politely declines the offer of the UWP XAML framework's help here, and sets the HighContrastAdjustment property of the app's main Page to be "None".

Figure 9: The Sa11ytaire app with switch device control on, showing system colors from the active high contrast theme for button text and background, disabled controls, and highlighted UI.

Windows Display Scaling

This is another situation where no real work has yet been done in the app, but work needs to be done. The display scaling feature in the Windows Settings app is an incredibly important feature. To increase the size of everything shown in the app, can help to deliver an easier-to-use app experience for many people, and sometimes an experience that's usable at all. But that assumes that an app accounts for the space it has available to show stuff. The screenshot below shows the app running at a display scaling of 300%, instead of the 200% that's the default value for the device that I'm using now. And the results are a mess. The text and suit symbols shown on the card are truncated. The app today does change the width of the dealt card lists to account for the width of the app, but it doesn't take action to resize or otherwise adjust the contents of the cards.

And this raises the same question that most apps have when considering what to do when there's just not enough space to present the information that the app wants to present. For example, should the app reduce the amount of information shown, by adding ellipses on truncated lines of text? Or perhaps the information is so important that it must be shown in its entirety, and so text will wrap to lie across multiple lines. All the Sa11ytaire app does today is put the entire set of dealt card lists inside a ScrollViewer. By doing that, at least all cards can be reached, even if not all the cards can fit on the screen at the same time. While this might make the app technically usable at the display scaling of 300% on my device, there's no way I would claim this is a good experience.

So this is another interesting topic for a later release.

Figure 10: The Sa11ytaire app showing truncated suit symbols and truncated text when a higher display scaling is active.

Windows Magnifier

The app worked with Windows Magnifier by default, due to the use of standard controls in the app.

Perhaps the most interesting aspect of Windows Magnifier's interaction with the app relates to bringing the element with keyboard focus into view when the element receives keyboard focus. Since all interactable elements in the app are based on standard controls, when keyboard focus moves to an element, UWP XAML raises a UIA FocusChanged event on behalf of the app. UIA clients like Windows Magnifier, (and the Narrator screen reader,) can then react to those events, and learn of the bounding rectangle associated with the element that has gained keyboard focus. And once Magnifier knows that, it can change its magnified view to present the element with keyboard focus on the screen.

Figure 11: A magnified view of a portion of the Sa11ytaire app, with Windows Magnifier in its fullscreen mode, and at a magnification level of 300%.

Narrator screen reader

The Narrator screen reader that comes with Windows, is a UIA client app. Narrator is absolutely dependent on the programmatic representation of the app, as exposed through the UIA API. Because all the UI in the Sa11ytaire app is based on standard controls, the app has a great head start on supporting the Narrator experience. For example, as keyboard focus moves around the app, Narrator can respond to the UIA FocusChanged events that are automatically raised, and can announce details of which element has gained keyboard focus.

Important: The app uses event handlers on its controls, which are standard for the type of control. For example, it has a Click event handler on a Button, a Checked event handler on a ToggleButton, and a SelectionChanged event handler on a ListView. By using standard event handlers on the controls, the UWP XAML framework automatically adds support to have the same functionality programmatically accessible through UIA. For example, when using Narrator on a touch device, Narrator's double-tap gesture can be used on the "Next card" element to invoke that button. And the double-tap gesture can be used to select a card to be moved, and then to specify where it should move to. If an app uses its own custom pointer event handlers to trigger action, then it's quite possible that the handlers won't work with Narrator on a touch device. Instead, it's likely that Narrator will eat the original pointer events while determining if a Narrator gesture has been issued, and those original pointer events won't make it through to the app.

While some support for Narrator will be available by default, it's important to verify that the app's UIA representation is a great match for the meaning of the UI. The Windows Speech Recognition section earlier mentioned how all elements should have an accurate, concise and localized UIA Name property. It's also very important for the Narrator experience to consider the LocalizedControlType property of the elements. For standard controls, UIA will automatically provide an appropriate LocalizedControlType, based on the element's UIA ControlType property. For example, a Button and ToggleButton both expose a LocalizedControlType of "button". But is that string the most helpful to people playing the Sa11ytaire game? To start exploring this question, the LocalizedControlTypes for some elements have been customized. For example, using strings such as "Remaining card pile", "Top upturned card", and "Target card pile".

Note: When first working on the app, the UIA representation of some elements was such that the default LocalizedControlType was exposed, custom friendly strings such as "Top upturned card" was set to be the UIA Name of the element, and the value of the card, (for example, "5 of hearts"), was set as a custom Value property. (Support for the UIA Value pattern can be added to a Button or ToggleButton through use of a custom AutomationPeer.) Technically this worked, but it didn't lead to a great experience when using Windows Speech Recognition (WSR). When using WSR, it should be possible to speak the UIA Name of an element and immediately interact with the element. When using WSR, it's natural to speak (say) "5 of Hearts" to interact with a card which visually shows "5 of Hearts". In order to enable that, the UIA Name property must match the value and suit of the card. So this was an interesting case where we needed to consider what single UIA representation can provide the best experience for a variety of UIA client apps.

Another interesting consideration here related to the UIA Help property associated with the elements. When I first started experimenting with the UIA representation, I was fairly liberal with adding Help properties which described the purpose of some elements. But now, I actually think all that information will become pretty irritating once the person using the app is familiar with the game, and so I expect I'll remove some of it. Having said that, it's critical that we provide some intuitive way to explain how to use the app. One colleague pointed out that at a minimum, some help content needs to be available, and must include the basics of the rules, and what keyboard shortcuts are available. And given that the color of the suit of the cards affects what moves are valid in the game, the help content also needs to describe which suits are black and which are red. It was also suggested that the UIA representation be updated to include the suit color, for people who are new to the game. I have to say, that's a really interesting idea. Perhaps we could add an option which would have the color included in the Narrator announcement. For example, "4 of clubs, black" or "King of Hearts, red".

After accounting for the above considerations, I expect the app would technically be accessible to someone using Narrator. So the next step is to consider how we can do better than that. What can we do to deliver a really efficient experience? And when considering this, the Windows 10 Fall Creators Update feature that really got our attention is the UWP XAML Notifications feature.

UWP XAML Notifications

By using this feature, a UWP XAML app can call AutomationPeer.RaiseNotificationEvent() to raise an event which requests that Narrator speak some arbitrary string supplied by the app. While this may seem a quite attractive thing for an app to want to do, inappropriate use of the event could quickly result in a grim experience for the person using the app. The person does not want to be bombarded with unnecessary announcements which could be at best irritating, or worse, impact their ability to complete their tasks. But due to the nature of the Sa11ytaire UI, it did seem at least worth considering whether the Notification event could help deliver a more efficient experience.

Another important aspect of the Notification event is that the app can specify how the event should be handled by a UIA client like Narrator, relative to other UIA events being raised by the app around the same time. Traditionally an app has had no control over that. For example, traditionally if an app raised a LiveRegionChanged event to make Narrator aware of a change in the app's UI, and Narrator received both that event and a FocusChanged event around that same time, it might be impossible in practice to get Narrator to make the announcement that the app intended. The new Notification event is designed to provide much more control to the app over that sort of experience.

The exploration into leveraging the Notification event, included the five cases listed below.

1. Announcing the current state of the three main areas of the app. (These announcements are triggered with presses of the F2, F3 and F4 keys.) For example, say keyboard focus moves to a card in a dealt card list, it would be tiresome to have to tab over to the upturned card pile or target pile, just to get a reminder of what cards are shown in those areas. So a press of F2 or F3 will announce the state of the remaining cards area or the target card piles respectively.

The following text is an example of what Narrator would announce in response to a press of F4, when the dealt card piles contain the cards shown in Figure 12.

"Pile 1: 7 of Diamonds to 10 of Spades, Pile 2: Queen of Hearts, Pile 3: 2 of Clubs to 3 of Diamonds, 2 cards face-down, Pile 4: Jack of Spades to Queen of Diamonds, 3 cards face-down, Pile 5: 4 of Spades, Pile 6: 2 of Hearts to 5 of Spades, 4 cards face-down, Pile 7: 3 of Hearts, 4 cards face-down,"

Figure 12: The Sa11ytaire app showing the dealt card lists with a mix of face-up and face-down cards.

2. Announcing the three cards shown in response to invoking the "Next card" button. For example: "Jack of Hearts, 5 of Clubs, 8 of Hearts on top".

3. Announcing what face-down card in a dealt card list became face-up as a card above it was moved away. For example: "Revealed 6 of Spades in dealt card pile 3". Note that this announcement would be made in such a way that Narrator would fully announce both it, and details of where keyboard focus has moved to as the card with focus moved to another dealt card list.

4. Announcing a hint on what moves are available. Overall the app's pretty much at a stage where the app can technically be played, and that's it. But Tim added an interesting feature that goes beyond that, and which actually offers some help for people unfamiliar with the game. The optional "Enable automatic hints" feature uses the Notification event to have Narrator automatically announce details of what moves are available between the dealt card piles, after a card has been moved. Given that the related announcement will be made around the same time as the "Revealed" announcement mentioned above, and also when keyboard focus moves, the Notification event would be raised in such a way that all these announcements would be completed by Narrator.

5. Announcing the switch device control highlight moving from one element to another. Given that this highlighting can be independent of an element gaining keyboard focus, by default Narrator would not react to the highlight moving. By using the Notification event, the person using the app can be made aware of the change in highlighting. This is another situation where app control of how the announcements should be managed is so important. The switch device control highlighting may be moving faster than Narrator can complete the related announcements. In that case, Narrator must interrupt itself to begin the next highlight-related announcement. To have the Narrator announcement lag behind where the switch device control highlight actually is, would result in the app being unusable.

Note: Tim also suggested that action should be taken by the app to have Narrator make an announcement when the state of switch device control is toggled. This is particularly important given that both the app and Visual Studio react to F10. If you're constantly switching between the app and the debugger, and a press of F10 only results in a Narrator announcement of "F10", it's not obvious at that moment whether the state of switch device control in the app has changed, or you've moved to the next line in the debugger.

Important: After experimenting with the Notification events for a while, we've not been able to achieve a consistent Narrator experience in the Windows 10 Fall Creators Update. That is, even when raising some events in such a way that we request that the related announcement must be made, sometimes it seems that the announcement is not made. However, it seems that the experience is more consistent with the latest Insider Builds of Windows 10. As such, we are not considering the Notifications events to be of interest in practice to the Sa11ytaire app experience until the next update of Windows 10. That said, if it all possible, we would like the use of Notification events to be a key part of delivering an efficient Sa11ytaire experience with Narrator, as soon as the next update of Windows 10 is available. In preparation for that, we'll continue to react to feedback on how these events might help to deliver the best possible experience.

A few more thoughts

At some point we'll add a Settings page to the app, but for now, all configurable settings are in the AppBar. The AppBar also includes a couple of buttons to launch the help and to restart the game. We set those buttons' AutomationProperties.AcceleratorKey properties to "F1" and "F5" respectively. By doing that, their UIA AcceleratorKey properties are exposed with those values, and Narrator can make the person playing the game aware of these handy keyboard shortcuts.

The ComboBox for selecting the switch device control highlight speed is available in the AppBar, and it is recognized that that UI itself is not accessible with the switch device. For the app to be really considered accessible to someone using a switch device, all its UI, including all settings, must be accessible. This is something we can explore further as we get feedback on this topic. (And as part of this, we can add more configurable settings relating to switch device use.)

By the way, the ComboBox uses AutomationProperties.LabeledBy to reference the TextBlock that provides the visual label for the ComboBox. By doing this, the UIA Name property of the ComboBox is set to the TextBlock's text string. The XAML for this is shown below:

<TextBlock x:Name="SwitchScanSpeedLabel" x:Uid="SwitchScanSpeedLabel" VerticalAlignment="Center" Margin="10 0 10 0" />

<ComboBox x:Name="SwitchScanSpeedComboBox" AutomationProperties.LabeledBy="{Binding ElementName=SwitchScanSpeedLabel}" Width="100" VerticalAlignment="Center" Margin="10 0 10 0" />

We do currently have a couple of settings relating to the Notifications event raised by the app through RaiseNotificationEvent(). These settings were helpful during development of the app, particularly when Narrator wasn't running. The "Show most recent notification" setting, adds the latest notification string to a read-only TextBox near the bottom of the app, and the "Notification audio" setting sends the notification string to the default text-to-speech (TTS) engine on the device.

Now, that last bit raises an interesting point. If the "Notification audio" setting is on, then the TTS engine will output audio while a screen reader might be making announcements at the same time. That leads to a poor audio experience, and a couple of other settings in the AppBar can be also affected by this. There's the "Enable automatic hints" settings which uses the Notification event to have Narrator automatically describe any available moves between the dealt card piles, after a move has been made. And while use of the Notification event allows control over how related screen reader announcements should mesh with other announcements from the screen reader, it can't account for strings being independently sent to a TTS engine.

And that brings us to the final setting in the AppBar, "Sound effects". That setting enables a success or failure audio notification following an attempt to move a card. This audio can be useful for such cases where you attempt to move a card, but the card didn't move. In that case, was the attempted move not valid, or did the target element not get toggled or selected as intended? But this audio is sent directly to the TTS engine, and so can effectively overlap with whatever Narrator's currently saying. So this is another opportunity for us to explore how an app might provide helpful success/failure audio feedback, regardless of whether a screen reader's being used.

And we also recognize that the "Sound effects" feature is inaccessible to someone who's deaf. For example, when an attempt is made to move a card to somewhere it can't be moved, there's no visual cue that the move was invalid. Given that we felt some feedback might be helpful in that situation, that feedback must be accessible to everyone playing the game.

All in all, I'm very excited about all the app can do today, and all the ways that people can interact with it, but there's a great deal still to explore. As the title of this post says, all we've done so far is set the scene.

Summary

The Sa11ytaire app has been a fascinating project to work on, with all the considerations around how the app can be built to be as accessible as possible by default, and then to take this further to deliver the most efficient experiences for all types of input and output methods that are available to a Windows Store app. The app is usable in many ways already thanks to the collaboration and feedback of my colleagues, and I really feel this is just the tip of the iceberg. I can't wait to learn how far we can take this.

And of course, if based on how you interact with your device, you feel that the app could be updated in some particular way to become more efficient for you, just let us know!

Guy

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Introduction

A few years ago I was in contact with an organization which works with kids with severe disabilities, many of whom are non-verbal, and I suggested that I build an app which could have custom text spoken by the computer when the student clicks on a button with whatever type of input device they were using. I'd already built many exploratory apps that included that functionality, and I knew .NET and Visual Studio made it practical for me to build this sort of app quickly. The goal would then be to update the app to make it as useful as possible for the students. For example, some of the students have low vision, and so I felt it might be helpful for the various elements in the UI to be able to appear with customizable colors and sizes.

Some of the visual customizations I made to the app would have been straightforward to make if I'd built the app as a UWP XAML app. And to have done so would have meant it would be simple for me to get the app up at the Microsoft Store and to manage updates to it. But when I first built the app, a number of people who would use it had machines running on Windows 7. Because of that, I built a WinForms app which could run on both Windows 7 and Windows 10 computers.

Before long, I had a communication tool, whereby the student could click on a button to have a specific announcement made. And over time the app evolved into a board maker app, where at any given time, it would present a set of buttons and when a particular button was clicked, the app would make some announcement and then would update its UI to show a different set of buttons. What's more, in addition to text being shown on the buttons, an image or video could be shown, and audio could be output when the button is clicked. All this functionality is practical for me to achieve thanks to the power of .NET. A collection of features available in the app is shown at 40 things to do in the free Herbi Speaks board-making app.

While working on this app, I was reminded of a very important point:

An app doesn't have to do much to be helpful – it just needs to do what's needed.

Many developers would consider the app in its earlier forms to have too few features to be helpful. And even with its larger set of features, it might be considered too unintuitive or just plain unprofessional to be something that anyone would want to use. And certainly, there are many ways in which the app could be improved, and it does have constraints that have led to questions from some customers. And something I thought long and hard about is the fact that the app's not accessible. For example, it can't be used only with a keyboard. But that said, I had to ask myself, is it better to make the app available to people, than not to? Should I hold off on making the app available until I've sorted out all the potential improvements, or should I share it with people who would find it helpful as it is, and prioritize improvements based on feedback? In the end I decided to share it out.

The screenshot below shows a demonstration Herbi Speaks board with three buttons on it. One button shows a question mark, and when that button is clicked, the app says "Where is Jasmine?". Another button shows a sheep, and the last button shows Jasmine the spaniel. Typically for such a board, when the Jasmine button is clicked, the app would make a helpful audio announcement or perhaps play a video of Jasmine, and then refresh to present a different board. Note that the app is running in its fullscreen mode, meaning that there is no other UI presented which the person using the app can interact with.

Figure 1: A demonstration Herbi Speaks board showing three buttons.

Was it worth building the app?

Yes. The app's being used at around 1:30 mins in Herbi Speaks in action, and while I clearly had a bug to fix in the app at the time, the video did show me how the app could play a useful part in the student's education.

I also received the following feedback from the mother of another person who uses the app.

"The Herbi Speaks app has been very helpful for us. The personalized boards we can make ourselves, with pictures and sounds our son relates to and really enjoys, ….

There are limitless topics we can use to interest him. We only need to use our imagination. He much prefers sounds attached to the images to really keep him interested. For instance, I made 30 boards all linked together of fish and other animals that live in water. They start out very simple with just one fish in the middle of the blue screen. The fact that we can pick the colors we want on everything is great! As he goes along I add in water plants on the screen to make it a bit harder to find the fish. I downloaded the fish/animal images from google search as well as water sounds that happen after the text speaks. Number 10 is a whale that has a nice long whale song. Toward the end is a dolphin and the sound it makes.

My severely disabled son has learned to breeze through all 30 different boards in 10 to 15 minutes. The challenge for us is in finding what does interest him. When we make a hit it is amazing to see what he can do. I have made many boards now on different subjects. Our goal of course is to get him to the point where he can actually communicate his wants, needs and feelings to us. Right now he is just having fun playing on the computer and learning what he can do."

I find this sort of feedback extremely helpful. And while I'm pleased that the app is proving to be of some use as it is today, it does make me wonder how I can make it more useful to this family. Is it practical for me to make the app more helpful as a communication tool for the person using the app?

And this really brings up the crux of all this. Many developers reading this will feel they could build an app like Herbi Speaks, or in fact something much higher quality. And I'm sure they could. So is it practical for you to build a simple app, with a very focused feature set, which helps plug a gap in what's available to someone today? If you do, I'd strongly recommend that you consider sharing what you've built with the world through the Microsoft Store. In my experience, leveraging the Microsoft Store is way more pleasant than messing around with my own app setup processes.

The following discussion relates to my experiences with making the Herbi Speaks app available at the Microsoft Store.

Making the WinForms app available at the Microsoft Store

I'd recently made another WinForms app available at the Microsoft Store, as described at Sharing assistive technology through the Microsoft Store, so I took a look at my comments there first, to reduce the chances of me missing a step. And below is how things went working on getting this second app up at the Store.

Preparing the app for submission to the Store

In Visual Studio, I added a Desktop Bridge Packaging Project to the solution containing my WinForms project. I already had a version of Visual Studio which supports packaging projects. When doing the "Add New Project" thing, it's "Windows Application Packaging Project" beneath Visual C# -> Windows Universal.

I then referenced the WinForms app in the packaging project, by right-clicking on the packaging project's Application item, then Add Reference, and then selecting the WinForms app.

When I then tried to build the packaging project, I did get a couple of errors. One error related to a mismatched processor type between the WinForms app and the packaging project. I'd hit that before, when preparing the other WinForms app for the Store. In both cases I addressed this by going to the Properties for the WinForms project, then Build, and changing the Platform Target from what I happened to previously have it set as, "x86", to "Any CPU". The other build error I got related to the version of the .NET Framework I was targeting in the build. It turned out I was targeting .NET 4, which I'd probably selected when I first built the app. I changed that to .NET 6.1, as I expect my customers will already have at least that on their machines now, and then the packaging project built fine.

Then it was a case of preparing the package for the Store. When I first considered the idea of moving the app to the Store a while back, I reserved the app name at the Store, so I didn't have to do that now. But I did have to associate the new packaging project with that reserved name. I did that by right-clicking on the packaging project, selecting Store, and then "Associate App with the Store".

My next step was to have the app package that I was building verified as being ok to upload to the Store. And sure enough, when I ran the verifier after building the package, I was told that the verification failed. I was told that the package was using the placeholder image assets. I tripped up on exactly the same thing when I worked on my previous app, and you'd think I'd have learnt, but apparently not. So again, it was really handy to get this early notification, rather than it being detected further down the line.

And having hit that, I then decided I'd try to be a little more proactive rather than reactive for a change. Last time I went through this process, my app wouldn't run after deploying from the package. That was because I'd built my WinForms app in such a way that an image would be required at run-time, but the image wasn't available to the exe after it was deployed. I was pretty sure that there wouldn't be an issue this time relating to images, but I was wondering about a couple of interop dlls that the app used. Would they be included in the package as the app needed them to be? Well, that turned out to be just fine. So long as my WinForms app is set up with the related assembly in the list of app references, and that assembly has a Copy Local property of True, then the packaging project will automatically include what's needed in the package. And I could verify that that was the case by deploying the Packaging Project on my machine, and making sure the app ran as expected. Nice one Visual Studio!

Submitting the app to the Store

It was now time to go through the steps of submitting the app to the Store. This started off being routine, but it was interesting for me when I got to the checkbox that said "This product has been tested to meet accessibility guidelines". I left that box unchecked, given that I know the app needs to be enhanced in a number of ways before I'd call it accessible.

I then submitted the app to the Store, and before long got told that it had failed certification. There were two reasons for that. The first is that I'd not supplied a link to a Privacy Policy. Such a link is required, because the app has access to my customers' user data folders. That's a requirement of the app, because my customers can associate images and media with buttons shown in the app, and the file browser in the app can take my customers to their Pictures and Music folders. And by default, the app also saves the file containing the data on the boards created in the app, in a folder beneath the Documents folder. So I created a privacy policy at my own site, and duly added a link to that when I later resubmitted the app at the Store.

The other reason why the submission failed certification was that the app had not gone through the full process required for submitting desktop apps at the Store, whereby I complete the form at Have you already converted your app with the Desktop Bridge? and then someone from the Store team works with me to confirm that the desktop app can be made available at the Store. I intentionally hadn't done that on this occasion, as I'd gone through that process with the previous app, and wasn't sure if it was required for all subsequent apps. But it turns out that I did need to go through the process for this second app. Fair enough.

This is where things got really interesting. After I'd supplied an app package to the person from the Store team for verification, he told me the app didn't start.

Make sure what you're supplying to the Desktop Bridge is good

When the person from the Store team let me know that the app didn't start, he sent over an event log which helped me learn that the error seemed related to my use of the .NET AxWindowsMediaPlayer control. (That's a really handy control, which makes it easy to add audio and video functionality to a WinForms app.) So as a test, I copied over my appx package to another machine with a recent version of Windows 10, and deployed the app there. To my surprise, the deployed app didn't start on that machine either. It turned out that Windows Media Player (WMP) wasn't available on that machine, and so I decided to assume that that was the same issue on the machine used by the person from the Store team. And it seemed reasonable to assume that if WMP isn't available on a machine, then my use of the AxWindowsMediaPlayer control could be impacted.

This is where I got distracted. I told myself that I should address this by updating my Desktop Bridge Packaging Project to have it set up the media-related components required by the app, if those components weren't already available on the target machine. So I would add whatever redistributable was required to the Packaging Project. I failed to find such a redistributable, but then someone pointed out that the issue here isn't the Desktop Bridge Packaging Project anyway. Rather, it's my WinForms app. The app isn't going to run on the target machine regardless of whether it's deployed through an appx package, or simply run as a WinForms exe. So I need to figure out what to do about my WinForms app first, and consider the Desktop Bridge after that.

So while it seems pretty obvious now, I should always make sure my WinForms story is rock-solid first. If I've got holes in that story, I shouldn't be trying to figure out how to plug those with changes to my appx package.

And then someone pointed out that WMP is available in Windows 10, but at some point it became an optional feature which is not available by default. So when the app's run, the AxWindowsMediaPlayer control may or may not be available. I don't know the recommended way of detecting whether the WMP feature's available at run-time, so I decided to add some exception handling around my AxWindowsMediaPlayer initialization. If I hit an exception, then I'll assume the problem is the feature's not available, and pop up a message to let the customer know what's going on. (Granted, I can't be sure that the exception is always related to the optional feature not being set up, but I'm hoping in practice it will be.)

The screenshot below shows the message now popped up by the app. It contains the following text.

The audio and video features of the Herbi Speaks app are not available on this computer. To enable these features, please visit the "Apps & features" page in the Windows Settings app. Then go to the "Manage optional features page" and add the "Windows Media Player" feature. When the Herbi Speaks app is then restarted, the audio and video features should be available.

Figure 2: A message box shown by the app when the app fails to initialize its media-related feature on startup.

In order to test this, I uninstalled the WMP optional feature on my dev machine. As an added twist to this, once I'd uninstalled the WMP feature, I couldn't build my app in Visual Studio. So I had to add the WMP feature through the Settings app to build my app, and then uninstall the feature to test my app. But after going back and forth a few times with adding and uninstalling the WMP feature, I'd updated the app such that my customers are made aware of how they can access the media features in the app, if those features are not available by default.

Note: For me, it was important to be using the Windows 10 Settings app to review the state of the WMP feature. The classic control panel also has "Turn Windows features on or off" functionality, but I couldn't find any reference to Windows Media Player there.

And so the app becomes available at the Store

My next step was to supply an updated appx package to the person from the Store team, who could then deploy and run the app, and perform the required verification steps. With his sign-off I could submit the appx package to the Store, and follow the standard steps for submitting any app. Given my learnings around the Windows Media Player, I added the following related details to the app's Release Notes which would appear in the Store:

If the Herbi Speaks app is running on a version of Windows 10 which does not have the Windows Media Player optional feature enabled, then audio and video cannot be played in the Herbi Speaks app. To enable audio and video in the app, enable the Windows Media Player optional feature from the Windows 10 Settings app.

And with that, the app soon became available at Herbi Speaks at the Microsoft Store.

Summary

There are many ways in which the Herbi Speaks app could be improved. For example, making it useable with a wider variety of input and output mechanisms. And giving the people who are setting up the boards some clue as to what to do when the app first runs, would be nice. But even with its current constraints, the feedback I've received lets me know that the app can be very helpful as it is today, and as such, I should make it available to as many people as I can. For me, this means making the app available through the Microsoft Store.

Please do consider whether it'd be practical for you to build a helpful tool which plugs a gap in what's available to someone today. And remember, the app really doesn't need to do much, it just needs to do what's needed. And if you do build such an app, consider sharing it with the world through the Microsoft Store.

Guy

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Important: Now that you know what the post refers to, you might well be thinking:

"Thanks very much Barker. I wasted my time visiting this post because I now know it doesn't apply to me. Couldn't you have included the purpose of the post in the title?".

Well, yes, I could. And that's the point of the post. Learn more…

Introduction

Many products present hyperlinks which provide an efficient mechanism for their customers to learn more about some topic pertinent to the product. And often the text shown on the links is such that the purpose of the link is made clear, even if the link were considered in isolation, and not as part of whatever other UI happens to be nearby. For example, I'm writing this in Word 2016, and went to the Options UI where I found a section on "Office intelligent services". That section contains a brief passage on what "Office intelligent services" are, and then presents a link which shows "About intelligent services". That link provides access to a much more in-depth description of the topic.

However, many products choose not to include topic-specific text in the text shown on the link. For example, UI might contain topic-specific static text, and follow that text with a link that only shows "Learn more". For customers who consume the link text and the nearby static text almost concurrently, this provides an efficient experience. But what about all your other customers? How would you feel if some of your customers who encounter the link have to take additional steps to learn what the "Learn more" link relates to?

For example, consider the experience for your sighted customers who use a mouse. You probably wouldn't ship a "Learn more" link that had no static text presented nearby describing the purpose of the link, and only present that purpose when the mouse is hovered over the link. While technically the purpose of the link is accessible to those customers in that scenario, it would be an inefficient and unfriendly experience, and you probably wouldn't ship it. In fact, you'd probably not want to ship an inefficient and unfriendly experience to any of your customers.

So let's consider the experience you're delivering to your customers who are blind, and who are controlling the Narrator screen reader through use of a keyboard. Say you present some descriptive static text visually and follow it in that visual UI with a link that says "Learn more". When your customers tab to the link, Narrator will announce "Learn more, link". While that's accurate, is the announcement as helpful as it can be? Wouldn't it be more helpful for it to say "Learn more about towhees", (assuming the current topic relates to towhees). For the link to only say "Learn more", forces your customer to take action to explore the nearby UI and figure out what the link's referring to.

What's more, screen readers often have a way to present their own UI which contains only the set of links in your UI. This usually makes it very efficient for your customers to interact with those links. But if your UI contains a bunch of paragraphs, all ending with links that say "Learn more", then your customer can get presented with a set of links in the screen reader's UI, and they all say "Learn more". That's a frustrating experience which you'll not want to deliver to your customers.

So by default, I'd recommend you make all the text on your links unique, and sufficiently informative so that your customer knows what the link refers to. This will provide an efficient experience whether tabbing through the UI and encountering a link, or reviewing the set of links presented in the screen reader's own UI.

But say your UI includes a Win32 SysLink or a WinForms LinkLabel. Both of those provide a way for you to include a text string in your UI, and for a particular subset of that text to be considered the interactive link. By default, when your customers using Narrator encounter a link where the link text is a subset of the full text, the UI Automation (UIA) Name properties associated with the links do not include the full text on the control, and so the Narrator announcement is not as helpful as it could be for your customers. This could lead to confusion or frustration for your customers. So the details below describe action you could take in your app to provide a more efficient, and friendlier experience for your customers who encounter those types of Win32 or WinForms controls.

Important: Never change your UI based solely on the experience with the Narrator screen reader. To do so would run the risk of degrading the experience with other assistive technologies, or even future versions of Narrator. Instead base your changes on potential enhancements to the representation of the UI through the UIA API. The UIA representation should be a clean, logical match for the meaning of the UI.

The Win32 SysLink

Say your .rc file contains a SysLink in a dialog similar to that shown in the following example.

IDD_ABOUTBOX DIALOGEX 0, 0, 270, 62

STYLE DS_SETFONT | DS_MODALFRAME | DS_FIXEDSYS | WS_POPUP | WS_CAPTION | WS_SYSMENU

CAPTION "About Grosbeak Translator"

FONT 8, "MS Shell Dlg", 0, 0, 0x1

BEGIN

    ICON IDI_GROSBEAK, IDC_GROSBEAK_IMAGE,14,14,21,20

    LTEXT "Grosbeak Translator, Version 3.1",IDC_STATIC,42,14,114,8,SS_NOPREFIX

    CONTROL "Grosbeak translation services are available in your area. <a href=""http://microsoft.com"">Learn more</a>",IDC_GROSBEAK_LEARNMORE,"SysLink",WS_TABSTOP,42,26,220,8

    DEFPUSHBUTTON "OK",IDOK,212,41,50,14,WS_GROUP

END

The following screenshot shows the visuals associated with the dialog when run.

Figure 1: A dialog box showing a SysLink control, whose subtext "Learn more" is presented as a link.

So next let's point the Inspect SDK tool to the dialog box. The screenshot below shows the hierarchy of the UIA elements associated with the SysLink. It reports that there's an element whose ControlType is Link, and whose Name is "Learn more". And it reports that this element is a child of an element whose ControlType is Pane, and whose Name is "Grosbeak translation services are available in your area. Learn more".

Figure 2: Inspect reporting the UIA hierarchy of the dialog containing the SysLink control.

What this means is as the customer using Narrator tabs around the UI, they'll encounter the link, and the announcement will be "Learn more, link". But learn more about what? This forces your customer to take additional action in order to learn the purpose of the link. This isn't the efficient experience I want to deliver to my customer.

Important: What we'll discuss below is how to customize the SysLink's UIA representation, such that a more efficient experience can be delivered. But it's not trivial to do that, and it would be great to avoid it if practical. So before taking any of that action, do consider whether it's practical to replace the use of the problematic SysLink with a link whose full, helpful text can be made the interactive link. For example, in the case above, make the interactive link have the text "Learn more about the grosbeak translations services available in your area". By doing that, your customer will know exactly what the link refers to when they tab to it, and you won't be spending time customizing the experience.

Ok, say for some reason you will be sticking with the SysLink with the subtext being the interactive part of the link. In this case we could consider customizing the Name of the Link element exposed through UIA to be the full SysLink text. While UIA Name properties should be concise, the full SysLink text doesn't seem too long to use as the Link's Name and would seem way more helpful if heard in isolation than "Learn more". In order to achieve this, we'll take two steps. The first is to get the text we want to set on the Link, and the second is to actually set it.

Getting the text of interest

We could add a new localized string containing exactly the text of interest, but as it happens the text we need is already available. It's the accessible name of the Pane element that contains the text. We can access that by calling AccessibleObjectFromWindow() with the SysLink hwnd, and then calling get_accName() to get the Name.

Setting the text on the Link

Having got the text, we can then call SetHwndPropStr() to set it on the Link element. However, we need to be very careful here to set it on the element we intended to. The hwnd involved with this UI is the SysLink hwnd, so we need to be careful not to set it on the Pane element, from which we just accessed the Name. To set the Name of the child Link element, I use the Inspect SDK tool again and note the LegacyIAccessible.ChildId property of that Link element. In this case the ChildId property has a value of 1, as shown in Figure 2 above. So I'll pass that value in with the call to SetHwndPropStr().

Note: I'm rarely interested in the LegacyIAccessible properties. I'm only interested in them here because the UI is Win32, and that was originally designed to support the legacy MSAA API rather than the UIA API.

By making the changes described above, I can then point Inspect to the Link and verify that its Name is now the more useful full SysLink text.

Figure 3: The Inspect SDK tool reporting that the Link element now has a UIA Name of "Grosbeak translation services are available in your area. Learn more".

And importantly, this means that when your customer tabs to the link, the announcement is "Grosbeak translation services are available in your area. Learn more". Your customer learns of the purpose of the link in an efficient manner, and so no more spending time trying to figure out its purpose.

Now, the above change alone is a helpful enhancement. But there is one more change we might consider. Now that the Link provides all the helpful information that the customer needs, the parent Pane element doesn't really serve much purpose. If your customers were to encounter that Pane element with the screen reader, then it would seem to be distracting duplicated information. So one additional change we could consider, would be to only expose that Pane element through the Raw view of the UIA tree. UIA has three views of its hierarchical tree. The Raw view contains all elements exposed through UIA. The Control view contains everything of interest to your customer, including all interactable controls and important static text. And there's also the Content view which I've never really paid much attention to. So if we expose the Pane element only through the Raw view, then the UIA tree that's exposed would seem cleaner.

The screenshot below shows the SysLink UI being represented only with a Link element through UIA's Control view.

Figure 4: Inspect reporting that the SysLink UI is now exposed through the UIA Control view as a single Link element.

We could consider taking this exploration even further. The UIA BoundingRectangle property of the Link element matches the location and size of the link part of the control. So a Narrator customer using touch input, would have to touch on that link part in order to hit the control. So perhaps we should set the BoundingRectangle property of the element to be the bounding rectangle of the parent Pane element. (I should say that I've never actually tried doing that.) But if we did that, a customer using the Windows Magnifier might tab to the Link, and find the magnified view change to bring the full SysLink into view, and that might not result in the keyboard focused link part being pulled into the view. This is an interesting consideration, but for now I'll leave the BoundingRectangle as it is.

Bonus tip: Since we're having so much fun polishing up the UIA representation, why not enhance the image element's representation while we're at it? The dialog contains an image of a grosbeak at a bird feeder. When customers using Narrator encounter the image, they'll hear "image" but not know what the image relates to. While the UI dev might feel that that's not an issue because the image really isn't that important, their customers won't know that. So rather than leaving your customers wondering if they're being blocked from important information being conveyed in the image, let's provide a more helpful experience. If the image was purely decorative, we could remove the UIA element from the Control view of the UIA tree in the same way that we did earlier with that outer Pane element. But in this case, let's say the image is conveying information that we want the customer to access, so we'll set an accurate, concise, unique and localized UIA Name property on the image, by calling SetHwndPropertyStr().

The screenshot below shows Inspect reporting the new UIA Name property on the image.

Figure 5: Inspect reporting that the UIA Name property of the previously nameless image, is now "A Grosbeak hanging on the side of a bird feeder".

Our original goal was to provide a more helpful experience when the customer using Narrator tabs to the link, and we achieved that by exposing a more helpful UIA Name property for the link. But having continued the exploration let's compare the announcements made by Narrator in response to the Capslock+W keyboard shortcut, relating to the main part of the dialog.

Before:

About Grosbeak Translator dialog,

image,

About Grosbeak Translator dialog,

Grosbeak Translator, Version 3.1,

Grosbeak translation services are available in your area. Learn more pane,

Learn more, link,

After:

About Grosbeak Translator dialog,

A Grosbeak hanging on the side of a bird feeder, image,

About Grosbeak Translator dialog,

Grosbeak Translator, Version 3.1,

Grosbeak translation services are available in your area. Learn more, link,

Overall this seems a more helpful experience.

The code I added to my Win32 project in order to make the changes described above is as follows:

// Near the top of the file...

#include    <initguid.h>

#include    "objbase.h"

#include    "uiautomation.h"

IAccPropServices* _pAccPropServices = NULL;

// When the UI is created...

HRESULT hr = CoCreateInstance(

    CLSID_AccPropServices,

    nullptr,

    CLSCTX_INPROC,

    IID_PPV_ARGS(&_pAccPropServices));

if (SUCCEEDED(hr))

{

    // Get the hwnd for the SysLink control.

    HWND hWndSysLink = GetDlgItem(hDlg, IDC_GROSBEAK_LEARNMORE);

    // Do not expose the Pane element through either UIA's Control view or Content view.

    VARIANT varView;

    varView.vt = VT_BOOL;

    varView.boolVal = VARIANT_FALSE;

    hr = _pAccPropServices->SetHwndProp(

        hWndSysLink,

        OBJID_CLIENT,

        CHILDID_SELF,

        IsControlElement_Property_GUID,

        varView);

    if (SUCCEEDED(hr))

    {

        hr = _pAccPropServices->SetHwndProp(

            hWndSysLink,

            OBJID_CLIENT,

            CHILDID_SELF,

            IsContentElement_Property_GUID,

            varView);

    }

    // Get an IAccessible associated with the SysLink control.

    IAccessible* pAcc = NULL;

    hr = AccessibleObjectFromWindow(hWndSysLink, OBJID_CLIENT, IID_IAccessible, (LPVOID*)&pAcc);

    if (SUCCEEDED(hr))

    {

        BSTR bstrName;

        // Now get the Name property of the main element associated with the SysLink.

        // In this situation, if will access the Name for the parent Pane element,

        // and not the Name of the child Link element.

        VARIANT varChild;

        varChild.vt = VT_I4;

        varChild.lVal = CHILDID_SELF;

        hr = pAcc->get_accName(varChild, &bstrName);

        if (SUCCEEDED(hr))

        {

            // Now set the Name on the child Link element.

            hr = _pAccPropServices->SetHwndPropStr(

                hWndSysLink,

                OBJID_CLIENT,

                1, // Pass in the LegacyIAccessible.ChildId for the child Link element.

                Name_Property_GUID,

                bstrName);

        }

        SysFreeString(bstrName);

        // And finally, give the image a useful Name.

        WCHAR szImageName[MAX_LOADSTRING];

        LoadString(

            hInst,

            IDS_GROSBEAK_IMAGE,

            szImageName,

            ARRAYSIZE(szImageName));

        hr = _pAccPropServices->SetHwndPropStr(

            GetDlgItem(hDlg, IDC_GROSBEAK_IMAGE),

            OBJID_CLIENT,

            CHILDID_SELF,

            Name_Property_GUID,

            szImageName);

    }

}

// When the UI is destroyed...

if (_pAccPropServices != nullptr)

{

    HWND hWndSysLink = GetDlgItem(hDlg, IDC_GROSBEAK_LEARNMORE);

    // Clear all the properties we set on the controls.

    MSAAPROPID propsPane[] = {

        IsControlElement_Property_GUID,

        IsContentElement_Property_GUID};

    _pAccPropServices->ClearHwndProps(

        hWndSysLink,

        OBJID_CLIENT,

        CHILDID_SELF,

        propsPane,

        ARRAYSIZE(propsPane));

    MSAAPROPID propName[] = {

        Name_Property_GUID };

    _pAccPropServices->ClearHwndProps(

        hWndSysLink,

        OBJID_CLIENT,

        1, // Pass in the LegacyIAccessible.ChildId for the child Link element.

        propName,

        ARRAYSIZE(propName));

    _pAccPropServices->ClearHwndProps(

        GetDlgItem(hDlg, IDC_GROSBEAK_IMAGE),

        OBJID_CLIENT,

        CHILDID_SELF,

        propName,

        ARRAYSIZE(propName));

    _pAccPropServices->Release();

    _pAccPropServices = NULL;

}

WinForms

Say your WinForms UI contains a LinkLabel, and you've set its LinkArea property such that text showing "Learn more" at the end of the LinkLabel becomes the interactive link.

Figure 6: WinForms UI presenting a LinkLabel with its LinkArea property set such that the interactive link part only contains the text "Learn more".

Now let's point the Inspect SDK tool at the UI and consider how this is being represented through the UIA API. The screenshot below shows that there's a Link element, which is a child of a text element. This is a similar structure to the UIA representation of the Win32 SysLink discussed above, other than the outer element has a ControlType of Text rather than Pane, and very importantly, the Link element here has no UIA Name property at all.

Figure 7: Inspect reporting that the UIA Link element associated with the LinkLabel has no UIA Name.

This means that when the customer using Narrator tabs to the link, the announcement will only be "Link". Technically the customer could use Narrator's Scan or Item navigation modes to explore what other elements are near to the nameless link, and assume the link is associated with the descriptive Text element. But that's a really frustrating thing to force your customers to do, and so if we can avoid it, we should.

Note: You may be thinking at this point, "But this is a framework issue, why should we as app devs have to spend time accounting for that?". And if you are thinking that, I'd say that'd be a fair point. Who knows, perhaps the framework will be updated at some point to set some helpful UIA Name on the link, but in the meantime, your customers are going to be frustrated. So this is the sort of situation where I suggest at least considering whether it is practical to help your customers with an app-side change, and if it is, go for it!

Important: The first approach I'd consider here is to drop the use of the LinkArea in the LinkLabel. In the example above, perhaps the text on the LinkLabel could be changed to be "Learn more about the grosbeak translation services available in your area" and the LinkArea use removed. That would mean the UIA Name on the link is set from the full text, and would be announced when customers using Narrator tab to it. That really would seem the simplest way to help your customers here.

If for some reason you really want to stick to using the LinkArea, one approach to consider would be to present your own custom class, derived from LinkLabel. When an AccessibleObject for the custom class is requested by the system, you would provide a custom AccessibleObject, and that would provide whatever custom Name property you feel is most helpful. Sometimes when doing this sort of thing, it's really not much work because the base class of interest has an AccessibleObject which specifically relates to that class. But MSDN makes no mention of anything like a LinkLabelAccessibleObject at AccessibleObject Class, so I'll pick the closest available, the ControlAccessibleObject.

Because I'll be using an AccessibleObject which isn't specific to the LinkLabel, I will need to override a few more things in order to replicate the UIA representation of a LinkLabel with that of my custom class. Whenever I do this sort of thing, I point the Inspect SDK tool to a standard LinkLabel, and then to my custom LinkLabel-derived class, and update my class's AccessibleObject until the UIA representations of the two controls are in all the ways that matter, the same.

The screenshot below shows Inspect reporting that with my custom LinkLabel-derived class, the Link element now has the helpful UIA Name set from the entire LinkLabel text.

Figure 8: Inspect reporting that the UIA Link element now has a UIA Name of "Grosbeak translation services are available in your area. Learn more".

Interestingly, by doing this, the parent/child structure associated with the original LinkLabel has now been replaced with a single Link element. What's more, the UIA BoundingRectangle property of this element covers the entire text, which seems potentially useful for customers using Narrator with touch. This again raises the question of whether this is the best experience for a customer using Windows Magnifier and who tabs to the link, but overall, to have the Link now exposed with a helpful Name seems a big improvement.

Oh my goodness! The Order!

Whenever examining the UIA representation of a WinForms app, it's very important to consider the order in which the elements are exposed through the UIA tree. The order in which the elements get exposed is based on the order in which the controls are programmatically added to the form. So that's often not the same as the order in which the elements are visually laid out, and not the same as the tab order. And sure enough, the Inspect screenshot above shows that the order of the button, the link and the static text in the UIA tree, is the opposite of the visual, logical order. So if my customer uses Narrator's Scan or Item mode navigation to move "next" through the UI, Narrator will actually move backwards through it. This will not do, so following the steps described in the "Order of elements in the UIA tree" section of Considerations around the accessibility of a WinForms Store app, I updated the app to have the order of the elements in the UIA tree match that of the visual, logical order.

Figure 9: Inspect reporting that the order of the elements exposed through UIA now matches the logical, visual order.

The code I added to my WinForms project in order to make the changes described above is as follows:

// Create a custom class which will provide the helpful accessible name for the link...

public class LinkLabelWithCustomAccessibleName : LinkLabel

{

    protected override AccessibleObject CreateAccessibilityInstance()

    {

        return new LinkLabelWithCustomAccessibleNameAccessibleObject(this);

    }

    public class LinkLabelWithCustomAccessibleNameAccessibleObject : ControlAccessibleObject

    {

        private LinkLabelWithCustomAccessibleName LinkLabelWithCustomAccessibleName;

        public LinkLabelWithCustomAccessibleNameAccessibleObject(LinkLabelWithCustomAccessibleName owner) : base(owner)

        {

            this.LinkLabelWithCustomAccessibleName = owner;

        }

        // Set the accessible name of the link part of the LinkLabel to be the entire LinkLabel text.

        public override string Name

        {

            get

            {

                return LinkLabelWithCustomAccessibleName.Text;

            }

            set

            {

                LinkLabelWithCustomAccessibleName.Text = value;

            }

        }

        // The remainder of the action taken by the class is required to replicate the default

        // accessibility of the LinkLabel.

        public override AccessibleStates State

        {

            get

            {

                // Return a state that accounts for whether the LinkLabel currently has keyboard focus.

                return AccessibleStates.Focusable | // The link is always keyboard focusable.

                (this.LinkLabelWithCustomAccessibleName.Focused ? AccessibleStates.Focused : 0);

            }

        }

        public override AccessibleRole Role

        {

            get

            {

                // The element has a control type of a link.

                return AccessibleRole.Link;

            }

        }

        // Activate the link when the default programmatic action is taken on the link.

        public override void DoDefaultAction()

        {

            // Assume this LinkLabel does contain a link.

            this.LinkLabelWithCustomAccessibleName.OnLinkClicked(

                new LinkLabelLinkClickedEventArgs(this.LinkLabelWithCustomAccessibleName.Links[0]));

        }

    }

}

// Instantiate that class in the form designer, instead of a standard LinkLabel...

this.linkLabelWithCustomAccessibleName1 = new LinkLabelWithCustomAccessibleName();

// Make sure that when the controls are added to the form in the designer,

// the order in which they're added matches the visual, logical order...

this.Controls.Add(this.label1);

this.Controls.Add(this. linkLabelWithCustomAccessibleName1);

this.Controls.Add(this.buttonOK);

Summary

Wherever practical, an app dev needs to deliver an accessible experience to all their customers that's both efficient and intuitive, with as little dev work as possible. After all, we don't have enough time to do the things we need to do, so we can't be spending any time on anything that's not necessary. Ideally, you won't have to take the steps I described above to adjust the default experience when screen readers encounter your links. And often you can avoid such work by presenting links with unique text which describes the purpose of the link, and which doesn't involve the type of control which presents the link text as an imprecise subset of a larger string.

But if for whatever reason the default experience at your app isn't as your customers need it to be, consider using SetHwndProp() or SetHwndPropStr() to enhance the accessibility of your Win32 UI, or using a custom AccessibleObject for your WinForms UI. Note that the steps shown above apply to far more than just links. Many controls could be enhanced through this sort of action if you feel it would help your customers.

Learn more about enhancing the programmatic accessibility of your Win32 and WinForms apps

Guy

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

Introduction

Hey devs, here's a question for you…

If you felt that a software tool running on Windows could help someone you know, and that tool isn't available today, could you build it?

There are certainly valid reasons why the answer might be "No". For example, perhaps the technology available to you today just doesn't support what you need. Or maybe it does, but you really don't have time to build the tool, given everything else you need to get done during the day. But perhaps in some cases, the answer might be "Yes". Perhaps today's technology does support what you need, and by grabbing a few hours here and there, you could build a first version of the tool and try it out. And if over time you build on that, you might end up with a tool that's helpful to a friend or family member, and one that you could ultimately make available to everyone.

So I do think it's important for devs to consider whether there's an opportunity to plug gaps themselves in what's available to people today.

Such an opportunity came my way last year when I was asked if I could build a tool that helped people to follow keyboard focus and the text insertion point while they were working with apps in Windows. By the time I got this request, I'd already built some related functionality in earlier explorations into leveraging the Windows UI Automation (UIA) API, and so it was fairly straightforward for me to build a first version of the new tool. I described some of that work at A real-world example of quickly building a simple assistive technology app with Windows UI Automation.

The app certainly has some constraints, and I think that's fine. I'd much prefer to release an app that works well in many situations, and not in others, rather than not releasing an app at all. (That's assuming I make people aware of where it doesn't work well.) For example, the app can't highlight things on the Start menu, and sometimes it struggles to highlight things in regular menus. But in many other places in many other apps, it works well, and is worth getting in front of people who might find it helpful.

And that brings me to another important question. What's the best way to get a Windows app in front of people? For me, the answer is the Microsoft Store. I'd already made the app available through my own site at Herbi HocusFocus, but if I can make the app available at the Store, then I don't have to carry on maintaining my own installation stuff. Instead I can build it all and release an update to the Store directly from inside Visual Studio. And a lot more people around the world who might find it helpful get to learn about the app.

My understanding is that today, it's not possible for a traditional Microsoft Store app to leverage the UIA Client API. My app relies heavily on that API, and so the next question is – how did I make the app available at the Store?

Below I've described what it took to get my WinForms desktop app up at the Store. Overall it was a pretty smooth process. And I'd say that's an impressive thing made possible by the Microsoft Store team, given that there was no way I was going to read any instructions if I could avoid it.

Figure 1: The Herbi HocusFocus app highlighting a Checkbox in its own UI.

Hold on a moment - why does the app have to be for someone else?

That's a good point. I asked the question above about you building a tool that could help someone you know. So that would include a friend, a family member, and yourself. All technology should be accessible, including the tools used to build software products. For example, Visual Studio is becoming more accessible with each release, and so should be usable to developers who also use assistive technology tools such as on-screen keyboards, magnifiers and screen readers. If you're aware of issues with the accessibility of the latest version of Visual Studio, send the team feedback – they want to hear from you.

Making the app available at the Microsoft Store

Ok, I must confess, I did read some of the instructions. The steps listed at Package an app by using Visual Studio (Desktop Bridge) got me off to a great start. I did have to upgrade the free version of Visual Studio that I had on my machine in order to be able to add a Packaging Project to my Visual Studio solution, but once I had the required version, that worked fine. And I'd reserved the name of my app at the Store in the same way that I would for any app at the Microsoft Store.

There were only two things I had to react to when trying to rebuild my WinForms app for the Store. Visual Studio spotlighted these for me as it hit them, so they were quick to resolve.

1. My original app had a "More information" link, which when invoked would call Process.Start() with the URL for my web site. The Process.Start() call is not available in the Store version of the app, and so I removed the link. This isn't a concern for me, as the listing of the app at the Store has the link to my site.

2. I'd not replaced the placeholder Store assets with my own assets. It was really handy for me that Visual Studio checked for that. It would have been rather unimpressive if a bunch of placeholder images later appeared for my app. So I duly added my own images for such things as the app logo.

With the above changes made, I filled in the form at Have you already converted your app with the Desktop Bridge? and a while later I was contacted by someone from the Microsoft Store team who could help with the next phase. They asked me to supply the appx package for the app, so that they could run it.

This is where I did have an "Oh no…" moment. The person from the Store team got back to me saying that after I'd supplied the appx package, the app wouldn't start. Uh-oh.

And this is where I felt I really should have been paying a little more attention to what I was doing. When the app starts, it dynamically selects an image to load. It does this so that it can show an image that accounts for whether a high contrast theme is active. (I mentioned this action when I turned the app into a case study on accessibility, at Considerations around the accessibility of a WinForms Store app.) However, the way I'd built the WinForms app, the image files got copied out to wherever the exe was running, and got picked up from there. I doubt this was deliberate action on my part when I first built the app. In fact, if the app run ok after I first it, I probably didn't even consider where the images were. Given that I'd not taken action to ensure the images were available after the app was installed via the new appx package that I'd built for the Store, the images couldn't be found when the app started, and so the app didn't start.

So to address this, I updated the app to embed the images as resources in the exe. This meant the app ran just fine when installed via the appx package. Hopefully I'll not make that mistake again.

Following this, the person from the Microsoft Store team could get familiar with the app, and approved it for uploading to the Store. Sure enough, I could then upload the appx package to the Store in the same way as a traditional Store app. I also went through the rest of the traditional steps for making an app available at the Store, such as uploading screenshots. In my case I chose a screenshot of keyboard focus being highlighted by the app in Edge, and the text insertion point being highlighted in Word 2016.

Figure 2: The Herbi HocusFocus app highlighting where keyboard focus is in the Edge browser.

Figure 3: The Herbi HocusFocus app highlighting where the text insertion point is in Word 2016.

Next steps

One of my goals in making the app available at the Store was that I'd get more feedback on how the app can be enhanced to become more helpful in practice. As it happens, as soon as the app was available at the Store and downloaded to a device that it had not run on before, I learned that my custom color picker can sometimes not show all the colors available. On one device, only 12 of the 16 expected colors appeared. I don't know if some mix of that device's display resolution and scaling is related to the issue. And I also learned that on one device the app's highlighting in Edge didn't appear as expected after the page had been scrolled.

So I've some fun investigations and updates to be making over the coming weeks. I'd say this sort of thing isn't too surprising for early versions of apps once they become available to a wide range of devices. I'll look forward to getting more feedback on where the app can be made more helpful.

Summary

Overall the process for getting my WinForms app up at the Microsoft Store was straightforward, through the helpful instructions at Package an app by using Visual Studio (Desktop Bridge), and by Visual Studio drawing my attention to things I needed to address. This means I can now reach more people around the world who might find the tool useful, and I can update it based on their feedback. The tool is freely available at Herbi HocusFocus at the Microsoft Store.

I really do feel privileged to be in a position to do this sort of thing. I'm familiar enough with, and have access to, certain technologies such as Visual Studio and UI Automation, and it would be wrong for me to not at least consider whether it'd be practical for me to build a tool that might plug a gap in what's available to someone. Sometimes I can't help, but sometimes I can. Also, I put the source for one version of the app out at Herbi HocusFocus 2.0 source code, for anyone interested in learning how the app leverages the UIA API.

So please do consider whether you could build a tool that a friend or family member, or yourself, might find helpful. And if you do build it, consider sharing it with everyone through the Microsoft Store.

Guy