Most of the examples for DisplayUI just show the simple case where all the extra data parameters are NULL or 0, and don’t really explain what all those parameters are for.  Mostly that’s because these parameters are vendor specific (i.e., different SR engines may have different behavior); the following behaviors are specifically for the Microsoft Desktop SR engine.  Also, the valid values for the parameters depend on the pszTypeOfUI parameter.

For the Microsoft Desktop SR engine, the pUnkObject parameter should always be NULL.  There are no (interesting) valid non-null values for this parameter.

pvExtraData and cbExtraData can be non-NULL if pszTypeOfUI is either SPDUI_UserTraining or SPDUI_AddRemoveWord. 

When pszTypeOfUI is SPDUI_AddRemoveWord, then pvExtraData should be a standard null-terminated string containing the word to add to the lexicon.

When pszTypeOfUI is SPDUI_UserTraining, then pvExtraData should be a double-null-terminated string containing a set of training sentences.

Typically there should be 20-30 sentences to get good results.

Well, sort of.

UI Automation was first released with Windows Vista, and in Vista, the only UI Automation providers were in managed code. 

During Windows 7, the Accessibility team ported the UI Automation providers to native code, and created a (bi-directional) “bridge” to Active Accessibility clients.  (They also back-ported the native providers to Vista and released them as part of the Platform Update for Windows Vista.)

Even in Windows 7, however, the native UI Automation providers did not have TextPattern support for edit controls and richedit controls.  That was added in Windows 8.

If you use the PRON attribute for a phrase, you will also want to use the DISP attribute as well to specify a display form, or else the textual representation of the phrase (in the SPPHRASE structure) and the text feedback will use the phoneme set.    If you’re not providing text feedback (and you’re using an in-process recognizer), then this will not be a problem.

In particular, training the engine will not add new words to the vocabulary.

Training the engine from a set of wave files is useful only if you wish to recognize more audio from other wave files from the same speaker.  It’s not even terribly useful if you want to recognize live audio from the same speaker; the noise characteristics are often different, resulting in a different acoustic model.

If your problem is that you have a set of words that don’t exist in the default dictation vocabulary, then what you need to do is create a lexicon.

There are two kinds of lexicons that are interesting in this situation: User Lexicons, and Application lexicons.  (Engine lexicons are also available, but can’t be modified.)

The difference is that User lexicons are specific to the user profile; whereas Application lexicons are shared across users and profiles.  There is only one User lexicon per profile, but there can be an arbitrary number of Application lexicons.

Entries in the User lexicon override entries in the Application Lexicon, and entries in newer Application lexicons override entries in older Application lexicons.

Application lexicons are ‘write-once’ – that is, once the lexicon has been created, it cannot be modified.  Typically, the application lexicon is created at application install time, and removed on uninstall.

Windows Speech Recognition has built-in wizards to manage the user lexicon (programmatically accessible via ISpRecognizer::DisplayUI or ISpTokenUI::DisplayUI; the TypeOfUI parameter should be SPDUI_AddRemoveWord).

I’ll go into more detail on the lexicon methods in my next post.

The most influential SF movie of all time (referenced and ripped off by everyone from Kubrick to Besson) is finally available in a complete edition. 

Fritz Lang’s 1927 masterpiece has only been available in incomplete form (and for decades was only available in a heavily re-edited version).

Two years ago, curators at the Buenos Aires Museum of Film discovered a version of the film that ran 25 minutes longer than any other known print.

It has been restored and will be available on DVD and Blu-Ray in November.

I can’t wait.

Determining when the shared recognizer is OFF is easy enough; ISpRecognizer::GetRecoState will return SPRST_INACTIVE.

But figuring out the sleep state is much harder.  That’s because the shared recognizer is active (SPRST_ACTIVE_ALWAYS), but it’s not listening for all commands.

In Windows 7, there is a new interface that expose ‘categories’ (ISpRecognizer3). 

When WSR is sleeping, ISpRecognizer3::GetActiveCategory will return a non-null ISpRecoCategory of type SPCT_SLEEP. 

When WSR is listening, ISpRecognizer3::GetActiveCategory will return a null ISpRecoCategory. 

NOTE WELL:  This is only guaranteed to work on Windows 7.  It is not guaranteed to work on future versions of Windows.

Some sample code to get an ISpRecognizer3 interface from a shared recognizer stored in m_cpReco:

In order to make this work, you’ll need to have the Windows 7 SDK installed and set as default in Visual Studio.

Also, you’ll need to make sure that you have the following version numbers defined (typically in targetver.h):

#define WINVER 0x0601

#define _WIN32_WINNT 0x0601

This will enable the Windows-7 only interfaces like ISpRecognizer3.

ISpRecognizer2::SetTrainingState switches the recognizer into or out of training mode.

When you go into training mode, all that really happens is that the recognizer gives the user a lot more leeway about recognitions.  So if you’re trying to recognize a phrase, the engine will be a lot less strict about the recognition.

The engine doesn’t really do any adaptation until you leave training mode, and you have set the fAdaptFromTrainingData flag.

When the engine adapts, it scans the training audio stored under the profile data.  It’s the training code’s responsibility to put new audio files where the engine can find it for adaptation.

These files also have to be labeled, so that the engine knows what was said.

So how do you do this?  You need to use three of lesser-known SAPI APIs.  In particular, you need to get the profile token using ISpRecognizer::GetObjectToken, and  SpObjectToken::GetStorageFileName to properly locate the file.

Finally, you also need to use ISpTranscript to generate properly labeled audio files.

To put it all together, you need to do the following (pseudo-code):

Create an inproc recognizer & bind the appropriate audio input.

Ensure that you’re retaining the audio for your recognitions; you’ll need it later.

Create a grammar containing the text to train.

Set the grammar’s state to pause the recognizer when a recognition occurs.  (This helps with training from an audio file, as well.)

When a recognition occurs:

Get the recognized text and the retained audio (using ISpRecoResult::GetAudio).

QI the audio for IStream.

Create a stream object using CoCreateInstance(CLSID_SpStream).

Create a training audio file using ISpRecognizer::GetObjectToken and ISpObjectToken::GetStorageFileName, and bind it to the stream (using ISpStreamFormat::GetFormat and ISpStream::BindToFile).

Copy the retained audio into the stream object (using IStream::CopyTo) .

QI the stream object for the ISpTranscript interface, and use ISpTranscript::AppendTranscript to add the recognized text to the stream.

Close the stream, update the grammar for the next utterance, resume the recognizer, and repeat until you’re out of training text.

However, with some languages, even this will not prevent your compositions from being terminated.

I had a chat with the TSF developers, and they said:

This was sort of by design on XP to terminate non-EA compositions by default. The design changed in Vista, but the change wasn’t ported to XP because it went as a new feature, not as a bug fix.

The good news is that there’s a workaround for XP: your text service needs to add\set this value in the registry:

HKCU\SOFTWARE\Microsoft\CTF\CUAS\NonEAComposition=2

(2 means ‘enable’, all others or no entry – non-EA compositions will be disabled).

Hopefully others will find this useful.

In particular, you need to specify an audio source and a recognition engine.  The shared recognizer uses the defaults for both of these.

Most people figure out that they need to specify an audio source; more commonly (and I’ve tripped over this myself) they forget to specify a recognition engine.

And while there’s a nifty helper for the audio input in sphelper.h (namely, SpGetDefaultTokenFromCategoryId(SPCAT_AUDIOIN, &cpToken)), there isn’t a helper for the recognition engine.

So, here’s a helper function that returns the object token for the default shared recognizer:

These windows do not take focus because these windows could alter the state of the application that WSR is talking to.  For example, if you’re renaming a file in Windows Explorer, when the focus goes away, the file gets renamed.

Unfortunately, though, accessibility applications often have problems finding these windows, even though these windows fire the standard WinEvents (several types of windows are plain dialogs, for example).

In Windows 7, WSR fires a custom UI Automation event when it shows or hides one of its custom windows (Disambiguation Numbers, dictation scratchpad, Alternates, Spelling, Pronunciation, Switch Windows, or Launch Application).

To do this, WSR uses the IUIAutomationRegistrar interface.

IUIAutomationRegistrar::RegisterEvent defines a new event ID given a GUID and a name.  If multiple applications register an event using the same GUID and name, they all get the same event ID.  The event ID can also be used in a call to SetWinEventHook, if your application is using WinEvents.

Beware: all registrations are invalidated when you release the IUIAutomationRegistrar interface pointer!  So you must retain the interface pointer until you no longer wish to process events.  (MSDN doesn’t make this clear.)

So, if you want to find out when WSR is showing one of its custom windows, call IUIAutomationRegistrar::RegisterEvent with these two event declarations, and check for these events in your event processing code.

const GUID GUID_ShowEvent = { 
    0x3891149e, 0x7190, 0x47d0, {0xa5, 0x18, 0xca, 0x1c, 0xdb, 0x40, 0xf7, 0xe3}
  };
#define UI_SHOW      L"Microsoft.Speech.UI.Shown"

const GUID GUID_HideEvent = { 
    0x987a1c35, 0x597b, 0x4947, {0x9e, 0xf8, 0xe7, 0xa6, 0x81, 0x42, 0xfd, 0x11}
  };
#define UI_HIDDEN    L"Microsoft.Speech.UI.Hidden"