Silverlight 4 OOB - Handwriting Recognition & Text-To-Speech

I have been sitting on this sample for quite a while, but never found the time to get it ready for a post. Before I switch the focus of my blog over to Windows Phone 7 development, I wanted to get this out of the way first :-)

With Silverlight 4, we introduced an exciting new capability for Out-Of-Browser apps: Trusted Applications - one of the key new fatures of Trusted Applications is the ability to integrate with COM Automation interfaces on your Windows PC. What does this mean? This allows you to call into native components on your system that are either provided by the OS or installed by the user or system admin. On MSDN you can find some sample code for the canonical example that highlights the power of this feature: integrating Microsoft Office features into your Silverlight application.

In this blog post, however, I want to show some other cool things you can do with these new APIs on your PC: I want to use the systems handwriting recognizer as well as the systems speech synthesizer within my Silverlight application. Let's do the easy part first and take a look at how we get the speech synthesis done (in the attached sample, this plays back the recognized handwriting):

1	private void btnSpeak_Click(object sender, RoutedEventArgs e)
2	{
3	dynamic spVoice = AutomationFactory.CreateObject("SAPI.SpVoice.1");
4	spVoice.Speak(tbResult.Text);
5	}

In line 3 we create an instance of the SAPI.SpVoice (SAPI = Microsoft Speach API). Now we can just call the Speak() method (line 4) in order to have the desired text spoken by the machine. Make sure you turn on your speakers.

For the handwriting recognition part, the code looks a little bit more complicated, mostly for two reasons: The COM Automation API for Ink has some oddities & we need to convert the users strokes from pixel space into HiMetric. Here is the full code needed to implement this as extension method on Inkpresenter to have its writing recognized by the system's handwriting recognizer:

1	private const double HiMetricFactor = 26.46d; // = 1000 * 2.53 / 96
2	public static string Recognize(this InkPresenter presenter)
3	{
4	dynamic recognizers = AutomationFactory.CreateObject("msinkaut.InkRecognizers");
5	dynamic reco = recognizers.GetDefaultRecognizer(1033);
6	dynamic context = reco.CreateRecognizerContext();
7	dynamic inkCol = AutomationFactory.CreateObject("msinkaut.InkCollector");
8	dynamic ink = inkCol.Ink;
9
10	foreach (Stroke stroke in presenter.Strokes)
11	{
12	int[] packets = new int[stroke.StylusPoints.Count * 2];
13	for (int i = 0; i < stroke.StylusPoints.Count; i++)
14	{
15	packets[i * 2] = (int)(stroke.StylusPoints[i].X * HiMetricFactor);
16	packets[i * 2 + 1] = (int)(stroke.StylusPoints[i].Y * HiMetricFactor);
17	}
18	dynamic inkStroke = ink.CreateStroke(packets, null);
19	}
20
21	context.Strokes = ink.Strokes;
22	dynamic result = context.Recognize(null);
23	return result.TopString;
24	}

If you are a little lost, that's OK. Here is how/why it works: :-)

On line 1 we just define the conversion factor for pixels to HiMetric (the coordinate space used by the native handwriting recognizer on the Windows system). Lines 4-6 are all about getting us a new RecognizerContext, which is the primary interface for the native recognition engines. In lines 6-7 I am using a little trick to get a new empty Ink object. There is no Automation-compliany way to produce one, but I can create an InkCollector that owns an Ink object. Then I can steal it for my purpose. All of lines 10-19 are about converting the Silverlight Stroke data (from pixels) into native Stroke data (in HiMetric). Last but not least, in lines 21-23 we pass the Strokes down to the recognizer have have them recognized synchronously.

The full source code for the sample is attached to this post. When you run it, please be sure to run it as Out-Of-Browser app, because Trusted Application features aren't supported in the browser in Silverlight 4.

Enjoy! :-)

 

HandwritingRecognitionDemo.zip

Comments

• Anonymous
  September 22, 2011
  Have u got any idea related to text to speech conversion in WP7??

• Anonymous
  February 12, 2012
  hi thank you for this post..i just want to know if i can interpret this code in windows phone7..if not, can you help me about "automation" and "AutomationFactory" and how i can use them in WP7 thank you

• Anonymous
  September 12, 2012
  impossible in windows phone 7. wp7 doesn't support Automation Factory