Rob's Rhapsody

WP8 Speech – Exploring the MVVM pattern

Rob Chambers — Fri, 05 Jul 2013 14:32:36 GMT

Today’s post will be the first in a series discussing how I’ve used the MVVM (Model, View, View Model) pattern to build a full blown user experience for the “Search On” application I’ve previously blogged about.

If you haven’t read the earlier posts, you might want to take a few minutes and see where we left off.

Since those posts went live, I’ve completely re-written the application with a full blown UX.

And … It’s not just a sample application anymore, either. Its an actual, real, publically available application.

Go Search On is available in the Windows Phone Store as of today, and … It’s FREE!

Before we dive into the developer details, let’s take a look at the new features I’ve added:

Designed to be used with or without speech recognition

The main screen has a textbox for the site you’d like to search, and another for what you’d like to search for.
Either textbox can be filled with the soft keyboard or using speech via a buddy microphone icon.
The site to search textbox has autocomplete, based on the same site list that we use to populate the voice command’s phrase lists.

Demonstrates the “dialog” and “implicit confirmation” speech UX metaphors

After users issue a Go Search On voice command (e.g. “Go Search On Amazon.com”)
… the app prompts and collects the search terms,
… then, the app informs the user (both visually and verbally) what it’s about to do.
… e.g. “Searching on Amazon.com for Legos”
While the “confirmation” prompt is playing, the user can either…
… do nothing, and the search will take place once the prompt is complete, or
… cancel the task by clicking cancel, or by attempting to edit the site name or the search terms

Search more “sites”

I’ve updated the list of sites available to all users, and added the ability for users to add their own personalized list of sites to search.
I’ve also added the ability to search the Windows Phone App Store as well as Xbox Music Marketplace,
… using the phone’s built in UX.

User authentication to allow CRUD (create, read, update, delete) type operations on what sites are available to search on:

Authenticated users can add, edit, and delete their own personalized list of sites to search.
The authenticated “Admin” has the ability to update the list of sites that is common for all users.
Authentication is performed via the Windows Live SDK, allowing users to use their Microsoft Account credentials, where …
… the Live Authentication Client Session’s authentication token is used to authenticate with Azure’s Mobile Service.

OK … Now, on the developer aspects of “Go Search On” …

Go Search On – Exploring the MVVM pattern with WP8 and Speech

The original sample application in the first post was only 100 lines of code. By the time we got to latest post, the app had more than tripled in size, and now … it’s much, much larger. That’s expected, though. Right? I mean, come on! Did you see that list of enhancements above?! :-)

So … To make the act of describing what’s going on in the new app more understandable and manageable, I’ve broken it down into the following pieces:

The basics – MVVM and Go Search On in it’s most simplistic high level form

The Model, View, and the View Model
XAML binding, Property change notifications, and Commands
Three different app interactions, with the same end result

And the rest …

Dictation into a textbox using MVVM Commanding – I’ve created a handy helper class you should be able to re-use in your own apps.
XAML Value Converters – what are they, why use them, and a few helper classes I’ve created.
Implicit Confirmation UX – what is it, and how I’ve done it in an MVVM manner.
Authentication – Using Windows Live, integrated with the Azure Mobile Service, demonstrating connections into the MVVM world.
Autocomplete – Demonstrating how I’ve done it in an MVVM manner, sharing the underlying models.
Site Lists – Exposing the site lists for All Users and the Current User using MVVM.
CRUD in MVVM – Implications for a basic MVVM based UX managing data in the cloud.

Today, we’ll tackle the basics, and I’ll try to tackle the rest of the topics in future posts.

Go Search On and MVVM Basics

If you’ve already built a few Windows Phone applications, you probably have seen or at least been exposed a bit to the MVVM concept. Many of the samples on MSDN, and the Visual Studio templates are built using XAML and MVVM. If you have absolutely zero MVVM exposure, I’d recommend watching a recent post/video on the Windows Phone Dev Blog, here.

Here’s my take on MVVM …

The View

In a Windows Phone MVVM application, the View is the collection of all XAML UIElements in a page, along with all associated code behind.

Once I got the hang of MVVM myself, I started to see that the code behind file got smaller and smaller over time. In fact, in my personal projects, I typically try to reduce the code behind file to almost nothing, other than the standard boilerplate code automatically generated by Visual Studio.

In Go Search On, the MainPage class in MainPage.xaml and MainPage.xaml.cs represent the view used when searching a site for a string of text.

The View has 3 UI elements we’ll discuss in today’s post:

The “[what site?] AutoCompleteTextBox,
The “[what?] TextBox, and
The “Go!” Button.

The Model

The Model consists of the data structures and all related classes, properties, and methods that operate independently from the View and/or the View Model. This independence is important for multiple reasons, most notably, sharing code between different platforms. The Model will likely be the most portable aspect of your entire project.

For my TiVo Command application I’ve mentioned before, I use the TivoCommand model in my WP app I’ve published in the store, in my W8 modern app I’m developing, and in my 64-bit console app running on Win7 (the utility that started it all). The Model is shared, 100%, between all three projects, even though, they have very different user interfaces (WP, Windows 8, character mode console app).

In Go Search On, the SearchOnModel class in Models\SearchOnModel.cs is the model that we’ll use when searching a site for a string of text. If you look through the SearchOnModel class, you’ll see a lot of familiar code from earlier posts.

There’s also a new public function called CreateNewSearchTask that takes a site name and a string of text to find on that site. I’ve restructured the earlier code to ensure all entries thru the “view” flow thru the same code path. You can think of this CreateNewSearchTask method as that funnel point. No matter how the user expresses their interest to search a particular site for something, the code flow will go thru this function.

The View Model

The View Model is where we tie together the View and the Model. You can think of the View Model as the UI-less version of the UI, that uses the underlying Model to do the heavy lifting. Essentially, the View Model represents everything that’s in the UI, in an object model that maps nicely to the UI, but with no actual UI code.

In Go Search On, the SearchOnViewModel class in Models\SearchOnViewModel.cs will help us do that “tying” together. The Model’s new function, CreateNewSearchTask, accepts two strings, whereas, on the View Model has two public string properties, SearchSiteName and SearchForWhat, which individually correspond to the two text boxes in our View. You’ll also find a public property called SearchNowCommand of type ICommand, that corresponds to the “Go!” button in the View (more on ICommand in a moment).

XAML Binding

Using Windows Phone’s Silverlight implementation of XAML Binding, we can tie together the View and the View Model using a special declarative syntax in our XAML markup. You can read more about Windows Phone’s XAML Binding and the Binding syntax here.

We want the public SearchOnViewModel’s SearchSiteName string property to be updated whenever someone updates the “[what site?]” AutoCompleteTextBox, and similarly, we’d like to keep the control up to date if we change the SearchSiteName programmatically.

We can achieve both of those goals by setting the Text attribute on the AutoCompleteTextBox XML element as follows:

<AutoCompleteTextBox Text="{Binding SearchSiteName, Mode=TwoWay}" ... />

Similarly, we can bind the SearchForWhat property on the View Model to the other TextBox in the View by setting the Text attribute on its XML element:

<TextBox Text="{Binding SearchForWhat, Mode=TwoWay}" ... />

Now, let’s hook up the “Go!” Button:

<Button Command="{Binding SearchNowCommand, Mode=OneWay}" Content="GO! " ... />

For the preceding three bindings to work, we also have to have set the DataContext for each control, or set it on a parent control, like this:

<Grid x:Name="LayoutRoot" DataContext="{StaticResource _model}">

Just as you’d expect, that “_model” is of type SearchOnViewModel, declared in the Resources section of our XAML page like this:

<so:SearchOnViewModel x:Name="_model"/>

That’s all we need as far as the XAML file is concerned for today’s post.

Now anytime a user updates the Text property on either the TextBox or the AutoCompleteTextBox, the View Model will be updated accordingly. Similarly, if we set the SearchSiteName or SearchForWhat string properties on the model programmatically, the UI controls will be updated appropriately. That’s what the “TwoWay” Binding Mode indicates. It goes both ways.

Once the user is happy with the text she sees in both controls, she can click the “Go!” button, and our SearchNowCommand will be invoked. That will eventually result in a call to the CreateNewSearchTask function on the Model, with the SearchSiteName and the SearchForWhat string properties as the arguments.

But wait… how does all that actually work? How do the controls get updated when the string properties are changed? How does that ICommand thingy work?

Good questions …

INotifyPropertyChanged

For binding to work properly in both directions, the public string properties can’t just be naked string properties.

Instead, you’ll need to implement both a setter and a getter, where the setter can notify subscribers when the property changes, via the INotifyPropertyChanged interface. When we use binding in the XAML, WP’s XAML implementation subscribes to the PropertyChanged event on INotifyPropertyChanged, and when we update the value, we’ll fire the PropertyChanged event.

Here’s what that looks like for our View Model:

   1:  public string SearchSiteName

   2:  {

   3:      set { SetSearchSite(value); }

   4:      get { return GetSearchSite(); }

   5:  }

6:

   7:  public string SearchForWhat

   8:  {

   9:      set { SetSearchForWhat(value); }

  10:      get { return GetSearchForWhat(); }

  11:  }

12:

  13:  private string GetSearchSite()

  14:  {

  15:      return _searchSite;

  16:  }

17:

  18:  private void SetSearchSite(string siteName)

  19:  {

  20:      _notifyHelper.UpdateValue(this, ref _searchSite, siteName, "SearchSiteName");

  21:  }

22:

  23:  private string GetSearchForWhat()

  24:  {

  25:      return _searchForWhat;

  26:  }

27:

  28:  private void SetSearchForWhat(string s)

  29:  {

  30:      _notifyHelper.UpdateValue(this, ref _searchForWhat, s, "SearchForWhat");

  31:  }

32:

  33:  private string _searchSite = "[what site?]";

  34:  private string _searchForWhat = "[what?]";

Now, you can see that I’m using a helper class to do the string updates, and the notifications. It’s very handy helper, and I’ve written it such that you should be able to use it for all your INotifyPropertyChanged implementations. I do… Here it is in it’s entirety:

   1:  public class NotifyPropertyChangedHelper

   2:  {

   3:      public event PropertyChangedEventHandler PropertyChanged;

4:

   5:      #region value setters

6:

   7:      public void SetValue(object sender, ref T value, T newValue, params string[] propertyNames)

   8:      {

   9:          value = newValue;

  10:          NotifyPropertyChanged(sender, propertyNames);

  11:      }

12:

  13:      public bool UpdateValue(object sender, ref T value, T newValue, params string[] propertyNames) where T : System.IComparable

  14:      {

  15:          bool fChanged = false;

16:

  17:          if (value == null || value.CompareTo(newValue) != 0)

  18:          {

  19:              fChanged = true;

  20:              value = newValue;

  21:              NotifyPropertyChanged(sender, propertyNames);

  22:          }

23:

  24:          return fChanged;

  25:      }

26:

  27:      #endregion

28:

  29:      #region notify helpers

30:

  31:      public void NotifyPropertyChanged(object sender, params String[] propertyNames)

  32:      {

  33:          foreach (string propertyName in propertyNames)

  34:          {

  35:              NotifyPropertyChanged(sender, propertyName);

  36:          }

  37:      }

38:

  39:      public void NotifyPropertyChanged(object sender, String propertyName)

  40:      {

  41:          PropertyChangedEventHandler handler = PropertyChanged;

42:

  43:          if (null != handler)

  44:          {

  45:              handler(sender, new PropertyChangedEventArgs(propertyName));

  46:          }

  47:      }

48:

  49:      #endregion

  50:  }

You’ll also have to hook it up in your View Model’s class, like this:

   1:  #region Common INotifyPropertyChanged properties and data

2:

   3:  public event PropertyChangedEventHandler PropertyChanged

   4:  {

   5:      add

   6:      {

   7:          _notifyHelper.PropertyChanged += value;

   8:      }

   9:      remove

  10:      {

  11:          _notifyHelper.PropertyChanged -= value;

  12:      }

  13:  }

14:

  15:  private NotifyPropertyChangedHelper _notifyHelper = new NotifyPropertyChangedHelper();

16:

  17:  #endregion

ICommand

When a XAML Button has an object bound to it’s Command property, and that object implements the ICommand interface, the underlying XAML implementation will not only fire the Click event when the button is pressed, it will also call ICommand’s Execute method.

Sounds like it’s time for another helper class, yes? I’ve got you covered with a new helper called Command, which implements ICommand for you.

When the SearchOnViewModel is constructed, I create a instance of that new helper class, specifying that when ICommand.Execute is called, I’d like the SearchNowCommand_Execute local class method to be called.

Basically, I just cache away the delegates passed in the constructor, and use them from the ICommand method implementations. Here’s what that looks like:

   1:  private void InitCommands()

   2:  {

   3:      _commandSearchNow = new Command(

   4:          "Search Now", (object param) => { return true; },

   5:          (object param) => { SearchNowCommand_Execute(param); });

And, for that to work, let’s take a look at the Command helper class, in its entirety:

   1:  public class Command : ICommand

   2:  {

   3:      public Command(Command command)

   4:      {

   5:          _execute = command.Execute;

   6:          _canExecute = command.CanExecute;

   7:          _name = command.Name;

   8:      }

9:

  10:      public Command(string name, CanExecuteStart canExecute, ExecuteStart execute)

  11:      {

  12:          _execute = execute;

  13:          _canExecute = canExecute;

  14:          _name = name;

  15:      }

16:

  17:      public string Name { get { return _name; } }

18:

  19:      #region ICommand interface implementation

20:

  21:      public event EventHandler CanExecuteChanged;

22:

  23:      public bool CanExecute(object parameter)

  24:      {

  25:          return _canExecute(parameter);

  26:      }

27:

  28:      public void Execute(object parameter)

  29:      {

  30:          _execute(parameter);

  31:      }

32:

  33:      #endregion

34:

  35:      #region delegates

36:

  37:      public delegate bool CanExecuteStart(object parameter);

  38:      public delegate void ExecuteStart(object parameter);

39:

  40:      #endregion

41:

  42:      #region private data

43:

  44:      private string _name;

  45:      private CanExecuteStart _canExecute;

  46:      private ExecuteStart _execute;

47:

  48:      #endregion

  49:  }

Again, I’ve tried to implement the Command helper class in Common\Command.cs in a way that you should be able to use it for most of your ICommand needs.

OK … Back to Go Search On …

Now that SearchNowCommand’s ICommand.Execute implementation calls thru to SearchNowCommand_Execute, I’ll need to make sure that we have a valid SearchSiteName, and a valid SearchForWhat. If we do, I’ll call off to another private method, SearchSiteForTextNow, passing the SearchSiteName and the SearchForWhat string properties.

If we don’t have a valid SearchSiteName, though, I'll use a Model method to get the search site via speech recognition (this is similar to what happened in previous posts when the wildcard/garbage rule was recognized in a voice command). Similarly, I’ll do the same for the SearchForWhat string property, if it’s still the default or if it’s empty.

Don’t worry about the method calls that set the visibility of some other controls, we’ll dig into those in a future post, although, it’s probably obvious what they’re trying to accomplish, maybe not just how they’ll accomplish it.

   1:  private async void SearchNowCommand_Execute(object param)

   2:  {

   3:      SetSearchNowCommandIsVisible(false);

   4:      SetCancelSearchNowCommandIsVisible(true);

5:

   6:      if (IsSearchSiteDefaultOrEmpty())

   7:      {

   8:          SetSearchSite(await _model.GetSearchSiteViaSpeech(true));

   9:      }

10:

  11:      if (!IsSearchSiteDefaultOrEmpty() && IsSearchForWhatDefaultOrEmpty())

  12:      {

  13:          SetSearchForWhat(await _model.GetSearchForWhatViaSpeech(GetSearchSite(), true));

  14:      }

15:

  16:      if (!IsSearchSiteDefaultOrEmpty() && !IsSearchForWhatDefaultOrEmpty())

  17:      {

  18:          await SearchSiteForTextNow(GetSearchSite(), GetSearchForWhat());

  19:      }

20:

  21:      SetSearchNowCommandIsVisible(true);

  22:  }

Now, finally, inside SearchSiteForTextNow, we call the CreateNewSearchTask method on the underlying Model.

   1:  private async Task SearchSiteForTextNow(string siteName, string findText)

   2:  {

   3:      SetSearchNowCommandIsVisible(false);

4:

   5:      try

   6:      {

   7:          _model.CreateNewSearchTask(siteName, findText);

3 user scenarios, 1 view model, 1 model implementation

Now, you might be wondering, why is it good to use MVVM. “This is a lot of code, Rob!!” Yes, you’re right!

To help understand why MVVM is a good thing, I’ll show you three different interaction flows the user may use to perform the same task:

User invokes app, fills in each field, clicks Go button
User invokes app, clicks Go button right away
User uses a voice command to invoke app (e.g. “Go Search On Amazon.com”)

Each one of these scenarios eventually wind up in the same location, and each of them always keeps the user interface up to date along the way.

In addition, in future posts I’ll cover several other aspects of MVVM in this app. After reading each of those, I hope the beauty of MVVM starts to become more clear. I can’t imagine not using MVVM for all the applications I build for WP from now on, except, perhaps, for the most trivial.

OK .. On to the scenarios…

User invokes app…

In all three scenarios, the flow starts with the user invoking the application.

The app invocation could be have been by a voice command, or normally.

From the earlier posts, you’ll probably recall that we tapped into the NavigationContext.QueryString from within the MainPage.OnNavigatedTo method to determine if the invocation was via voice command or not, and when it was not a voice command, we make sure that the Voice Commands were already installed. We’ll do the same type of thing in our new version of the app, like this:

   1:  protected override void OnNavigatedTo(NavigationEventArgs e)

   2:  {

   3:      base.OnNavigatedTo(e);

4:

   5:      if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))

   6:      {

   7:          HandleVoiceCommand(NavigationContext.QueryString);

   8:      }

   9:      else if (e.NavigationMode == NavigationMode.New)

  10:      {

  11:          HandleNonVoiceCommandInvocation();

  12:      }

  13:  }

Now, for all the non-voice command invocations, we’ll call off to the View Model from the View’s code to ensure that our application’s Voice Commands have been installed, and that the corresponding PhraseLists are all up to date. Like this:

   1:  private void HandleNonVoiceCommandInvocation()

   2:  {

   3:      _model.EnsureVoiceCommandsInstalled();

   4:  }

5:

Then, in the View Model, we’ll chain off to the underlying Model itself, where the real meat of the work takes place:

   1:  public void EnsureVoiceCommandsInstalled()

   2:  {

   3:      _model.EnsureVoiceCommandsInstalledAsync(new Uri("ms-appx:///vcd.xml"), false).Wait(0);

   4:  }

From there, it’s pretty similar to the code from before for installing the Voice Command Definitions, and making sure the PhraseLists are up to date.

Now … onto the specifics for the 3 scenarios…

User invokes app, fills in each field, clicks Go button

Here’s the basic flow of the app in this scenario:

Application starts, initializing the View (aka MainPage)

The View ensures that the View Model has been created
The View Model ensures that the Model has been created

MainPage.OnNavigated is called

App could have been invoked by a voice command, or normally
MainPage.OnNavigated determines invocation is not a voice command invocation
… Calls the View’s HandleNonVoiceCommandInvocation
… … calls the View Model’s EnsureVoiceCommandsInstalled
… … … calls _Model’s EnsureVoiceCommandsInstalledAsync
The Model ensures that the voice commands have been installed, and the Phrase Lists are all up to date (similar to previous post)

MainPage is Loaded

XAML binding updates …
… AutoCompleteTextBox.Text property from
… … View Model’s SearchSiteName string value (which defaultly is “[what site?]”
… TextBox.Text property from
… … View Model’s SearchForWhat string value (which defaultly is “[what?]”

User types in site name, clicks back to dismiss keyboard

AutoCompleteTextBox.Text property is updated
… XAML binding updates …
… … View Model’s SearchSiteName string value

User types in "for what" text, clicks back to dismiss keyboard

TextBox.Text property is updated
… XAML binding updates …
… … View Model’s SearchForWhat string value

User clicks on Go button

XAML Button click logic calls ICommand.Execute
… Calls the View Model’s SearchNowCommand_Execute method
… … calls the View Model’s SearchSiteForTextNow passing SearchSiteName and SearchForWhat
… … … calls the Model’s CreateNewSearchTask with passed SearchSiteName and SearchForWhat

User invokes app, clicks Go button right away

Now, what if the user just clicks Go right away instead of taking the time to fill the fields out?

Its very similar to the previous flow, except, we’ll make sure that the SearchSiteName and SearchForWhat properties are not empty and aren’t the defaults (see step 3 below). If they are blank or the default (see step 4), we’ll use the Model to ask the user via the WP8 speech APIs.

Application starts, initializing the View (aka MainPage)

The View ensures that the View Model has been created
The View Model ensures that the Model has been created

MainPage.OnNavigated is called

App could have been invoked by a voice command, or normally
MainPage.OnNavigated determines invocation is not a voice command invocation
… Calls the View’s HandleNonVoiceCommandInvocation
… … calls the View Model’s EnsureVoiceCommandsInstalled
… … … calls _Model’s EnsureVoiceCommandsInstalledAsync
The Model ensures that the voice commands have been installed, and the Phrase Lists are all up to date (similar to previous post)

MainPage is Loaded

Binding updates …
… AutoCompleteTextBox.Text property from
… … View Model’s SearchSiteName string value (which defaultly is “[what site?]”
… TextBox.Text property from
… … View Model’s SearchForWhat string value (which defaultly is “[what?]”

User clicks on Go button

XAML Button click logic calls ICommand.Execute
… Calls the View Model’s SearchNowCommand_Execute method, which
… … calls the Model’s _GetSearchSiteViaSpeech method and updates the SearchSiteName property
… … … which notifies the XAML binding, updating the AutoCompleteTextBox.Text property
… … calls the Model’s GetSearchForWhatViaSpeech method, and updates the SearchForWhat property
… … … which notifies the XAML binding, updating the TextBox.Text property
… … and finally …
… … calls the View Model’s SearchSiteForTextNow passing SearchSiteName and SearchForWhat
… … … which calls the Model’s CreateNewSearchTask with passed SearchSiteName and SearchForWhat

As you can see in step 4, simply because we set the View Model’s SearchSiteName and SearchForWhat properties with the speech recognition text results, the UI magically updated, without us having to care or remember.

User uses a voice command to invoke the app (e.g. “Go Search On Amazon.com” )

Now, conceptually, I consider a voice command as defined in the Voice Command Definitions file (e.g. vcd.xml) to be part of the View. There’s an underlying system that “renders” it, and does what’s needed, just like a Button.

Similarly, a command is a user’s request to perform some action, either via a XAML Button, or via a Voice Command. Now, with Voice Commands, as you’ve learned in earlier posts, we might need to pass query parameters that contain what voice command was recognized, and what items a user might have said from related PhraseLists. If only there were a way to pass a parameter to commands… Oh wait… There is … ;-)

ICommand.Execute receives a parameter of type object upon invocation, which can be null. For a Button, the parameter passed is the object that’s set or bound to the Button.CommandParameter property either via XAML or programmatically at runtime.

We’ll take advantage of that facility to make our Voice Commands look like Button Commands, passing the query string as the single parameter.

   1:  private void HandleVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      _model.VoiceCommand.Execute(queryString);

   4:  }

Then, in the View Model, we’ll funnel the voice command invocation down to the SearchSiteVoiceCommand_Execute method, similarly to how the Button click winds its way down into the SearchNowCommand_Execute method:

   1:  private void VoiceCommandServiceCommand_Execute(object param)

   2:  {

   3:      IDictionary<string, string> queryString = param as IDictionary<string, string>;

   4:      if (queryString != null)

   5:      {

   6:          VoiceCommandServiceCommand_Execute(queryString);

   7:      }

   8:  }

9:

  10:  private void VoiceCommandServiceCommand_Execute(IDictionary<string, string> queryString)

  11:  {

  12:      switch (queryString["voiceCommandName"])

  13:      {

  14:          case "searchSite":

  15:          case "searchSiteCatchAll":

  16:          case "searchSiteCatchAll2":

  17:          case "searchSiteCatchAll3":

  18:          case "searchSiteCatchAll4":

  19:          case "searchSiteCatchAll5":

  20:          case "searchSiteCatchAll6":

  21:          case "searchSiteCatchAll7":

  22:              SearchSiteVoiceCommand_Execute(queryString);

  23:              break;

  24:      }

  25:  }

26:

  27:  private async void SearchSiteVoiceCommand_Execute(IDictionary<string, string> queryString)

  28:  {

  29:      SetSearchNowCommandIsVisible(false);

  30:      SetCancelSearchNowCommandIsVisible(true);

31:

  32:      SetSearchSite(

  33:          queryString.ContainsKey("siteToSearch")

  34:              ? queryString["siteToSearch"]

  35:              : await _model.GetSearchSiteViaSpeech(false));

36:

  37:      if (!IsSearchSiteDefaultOrEmpty())

  38:      {

  39:          bool fVoiceCommandServiceAlreadyPrompted = queryString.ContainsKey("siteToSearch");

  40:          SetSearchForWhat(await _model.GetSearchForWhatViaSpeech(GetSearchSite(), !fVoiceCommandServiceAlreadyPrompted));

  41:      }

42:

  43:      if (!IsSearchSiteDefaultOrEmpty() && !IsSearchForWhatDefaultOrEmpty())

  44:      {

  45:          await SearchSiteForTextNow(GetSearchSite(), GetSearchForWhat());

  46:      }

47:

  48:      SetSearchNowCommandIsVisible(true);

  49:  }

Even though the SearchSiteVoiceCommand_Execute method is very similar to the SearchNowCommand_Execute, there are a couple key differences:

We need to set the SearchSiteName based on the Voice Command query string parameter named “siteToSearch”.

If we don’t have one of those, we’ll use Speech to get one, just like the 2nd scenario

When we try and get the SearchForWhat via Speech, we might not want to prompt.

If we hit the wildcard voice command case,
… we’ll collect the SearchSiteName, after the VCD TextFeedback prompted the user for that.
Then, we’ll have to prompt the user directly, ourselves, for the SearchForWhat text,
… with a prompt like “Search Amazon.com for what?”

OK. Now with that, let’s look at the entire app flow for this 3rd scenario:

Application starts, initializing the View (aka MainPage)

The View ensures that the View Model has been created
The View Model ensures that the Model has been created

MainPage.OnNavigated is called

App could have been invoked by a voice command, or normally
MainPage.OnNavigated determines invocation is a voice command invocation
… Calls the View’s HandleVoiceCommandInvocation
… … calls the ICommand.Execute for the ICommand obtained from SearchOnViewModel.VoiceCommand
… … … calls the View Model’s VoiceCommandServiceCommand_Execute method
… … … … calls the View Model’s SearchSiteVoiceCommand_Execute method

Meanwhile, MainPage is Loaded

Binding updates …
… AutoCompleteTextBox.Text property from
… … View Model’s SearchSiteName string value (which defaultly is “[what site?]”
… TextBox.Text property from
… … View Model’s SearchForWhat string value (which defaultly is “[what?]”

SearchSiteVoiceCommand_Execute continues

… … calls the Model’s GetSearchSiteViaSpeech method and updates the SearchSiteName property
… … … which notifies the XAML binding, updating the AutoCompleteTextBox.Text property
… … calls the Model’s GetSearchForWhatViaSpeech method, and updates the SearchForWhat property
… … … which notifies the XAML binding, updating the TextBox.Text property
… … and finally …
… … calls the View Model’s SearchSiteForTextNow passing SearchSiteName and SearchForWhat
… … … which calls the Model’s CreateNewSearchTask with passed SearchSiteName and SearchForWhat

The End!

Hopefully you now have a better idea how you can use MVVM with your WP8 application, and how you can hook up voice commands in this brave new MVVM world. You can download the full source for the project here. It will build, but won’t work without my secret keys for Azure and for Windows Live, but I can’t really share those, now can I?

And … You can download the app from the Windows Phone Marketplace here, as well.

I hope you find the app, the source, and the blog post useful. ;-)

WP8 Voice Commands: Dynamic Phrase Lists … From Azure!

Rob Chambers — Wed, 14 Nov 2012 17:02:21 GMT

Over the last couple weeks, I’ve been demonstrating various features of the new speech platform included in Windows Phone 8.0. So far, we’ve covered the following topic areas:

Start Speech Enabling your apps! … where I consolidated a bunch of resources, and uploaded a video of a “real world” Windows Phone application that I’ve “speech enabled” myself, called Tivo Command.
How many lines of code does it take to build a complete WP8 speech app? … where we found the answer to be a very small number. Especially considering the fact that our first version of our “Search On” sample app used Voice Commands, Speech Recognition, and Speech Synthesis.
WP8: Voice Command Phrase Lists … building upon our “Search On” sample app, we saw how we could extend the voice commands our users can use, simply by using elements inside our Voice Command Definition (VCD) xml file.
WP8: Voice Command Wildcards … we extended our app to deal with the cases when a user said something unexpected, by using a handy Voice Command Service feature, we call wildcards (we also sometimes call it a garbage rule, or garbage model)

Today, I’m going to pick up where we left off with our “Search On” application, and demonstrate how you as “the developer” behind the app, can update the phrases that users can say to your app, from the server side of the system.

The “speech part” of this is actually quite simple. In fact, if we had a list of strings (List) just laying around inside our app, it would only takes a few lines of code to update our Phrase List contained inside our VCD that’s loaded into the Voice Command Service. The harder part, truly, is getting that list from a server, hosting the list on said server, and dealing with other changes that result from the list being server side.

Since this is a predominantly “speech” blog posting, I thought I’d lead with the “speech” part of the code. If you’re one of those types that likes spoilers, feel free to scan ahead to take a look at the full solution, but I’ll try to make it worth your while to hold off until we get there. :-)

The “Speech” Part

OK. The “speech” part of updating phrase lists is pretty straight forward, like I said. Let’s look at the code…

   1:  private async Task UpdatePhraseListsAsync()

   2:  {

   3:      foreach (VoiceCommandSet cs in VoiceCommandService.InstalledCommandSets.Values)

   4:      {

   5:          List<string> updatedListOfPhrases = GetPhrasesForUpdatedSiteToSearchPhraseList(cs.Language.ToLower());

   6:          await cs.UpdatePhraseListAsync("siteToSearch", updatedListOfPhrases);

   7:      }

   8:  }

If you recall, a VCD file can have one or more elements, representing different “languages” or “dialects”. Our VCD file only has one, for “en-US” at the moment. I’ll probably post an update to our “Search On” sample app in the future, enabling voice commands in more than one language, and thus containing more than one . But for now, we just have one. Having said that, I’m going to go ahead and write the code for updating the phrase lists in a language neutral way, such that it should just work once that future post actually gets written.

To update the phrase list, as you can see, I’ve created a new method, called UpdatePhraseListsAsync, which simply iterates through all the installed Command Sets, gets a list of strings (the phrases users can say), and then calls UpdatePhraseListAsync on the VoiceCommandSet object we got from the VoiceCommandService.

One caveat I want to warn you about, though. Even if you’ve successfully installed your Command Sets in the VoiceCommandService, the InstalledCommandSets collection may be empty. That happens if you didn’t include a “Name” on the in your VCD file. We did this because the collection is actually a Dictionary, and thus, you have to have a “key” to look up the “value”. No key, No value. Thus, we have to update our VCD as follows:

   6:  <CommandSet xml:lang="en-US" Name="commands-en-us">

Great. The speech part is done. Let’s run it! … Wait. We have a lot of other changes. Like, getting the phrases for the siteToSearch phrase list. Let’s take a look at that method by method.

First up, GetPhrasesForUpdatedSiteToSearchPhraseList:

   1:  private List<string> GetPhrasesForUpdatedSiteToSearchPhraseList(string language)

   2:  {

   3:      List<string> phrasesBuiltIn = GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList();

   4:      List<string> phrasesFromAzure = GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(language);

5:

   6:      return phrasesFromAzure.Union(phrasesBuiltIn).ToList();

   7:  }

First, we get a list of phrases rom the built in list, then a list of phrases from Azure, and then we returned the combined list. Neat, eh?

Wait… Did I just say from Azure? Yeah. I did.

The “Hosting in Azure” Part

While I was considering where to store my list, I remembered that some of my friends over in the Azure team just released a really cool new feature called Azure Mobile Services. You can read about it here from the original announcement in August on ScottGu’s blog, or here from the Build announcement / blog post a couple weeks ago.

There are many things I love about Azure Mobile Services. But I’ll only list a couple here: They have a free 90 day trial; the To Do sample app on their quick start page makes it brain dead easy to build your first app with data hosted in azure; there are APIs for Windows 8, Windows Phone 8, and REST based APIs as well.

OK. You can either follow along here and use my Azure Mobile Service, (hosted here), or you can sign up for your own free trial, and have your own server side solution for your version of our “Search On” sample application.

Setting up your Azure Mobile Service

To set up my Azure Mobile Service. Here’s what I did, step by step (you can do the same, once you have an account):

Click on the Mobile Services tab:

Click “New” to create a new Mobile Service.

Click “Compute”, “Mobile Service”, and then “Create”. That should land you here:

Pick a URL (I picked robchblogsamples), select “Create a new database”, and pick your subscription. Then click the “Next” arrow.

Confirm the name of your new database, and then fill in the credentials you want to use for your new sql database. When you’re done, click the “Complete” check mark in the lower right hand corner.

That’s it! You now have your very own Azure Mobile Service.

I’d suggest you play with the WP8 To Do application for a few minutes, by clicking on the “Create a new Windows Phone 8” application link on the quick start tab you’re already sitting on … Once you’ve played around with that, come back here…

OK… Was that fun? Seems like you were only gone for a few seconds. :-)

Now, without going into a lot of details today, here’s the quick version of what I did to populate my Azure Mobile Service with a table filled with “rows” of data, that represent different “sites” that the user can search using our “Search On” application:

Create a new table, called SiteToSearchItem
Set the permissions on that table such that “Everyone” has read permission
Modify the WP8 To Do app you just played with, to add “sites” instead of To Do items
And finally … use that app to add some sites.

Make sense? OK … Let’s do that step by step:

Creating the table, setting the permissions

Create the new table, called SiteToSearchItem, by first, clicking on the “Data” tab… Then click the “Create” button at the bottom, and fill out the fields appropriately:

After clicking the “Complete” check mark in the lower right hand corner, your table will be created. It’ll just take a few seconds.

Now click your new “SiteToSearchItem” table. No data! Well … that makes sense. We haven’t added any yet. Let’s do that now.

Modify the WP8 app to add “sites”

Now, we’re going to modify the WP8 To Do app from the quick start tab in your new Azure Mobile Service, to add “sites” instead of “to do” items. At the highest level, here’s what I did to change that app in that way:

In MainPage.xaml.cs:

Renamed the ToDoItem class into a new SiteToSearchItem class.
Removed the two existing “To Do” data members from that class that were “To Do” specific.
Added public get/set properties for SiteName, Domain, SearchUrlTemplate, and Language.

Examples for those values would be:
SiteName = “Bing”, Domain = “www.bing.com”, SearchUrlTemplate = “http://www.bing.com/search?q={0}”, and Language = “en-US”

In MainPage.xaml:

Removed Checkbox, and To Do Textbox
Added Textboxes for SiteName, Domain, SearchUrlTemplate, and Language
Added a ScrollViewer, and Reformatted to make more usable

Here’s what that ended up looking like, in MainPage.xaml.cs:

   1:  using System;

   2:  using System.Collections.Generic;

   3:  using System.Linq;

   4:  using System.Net;

   5:  using System.Runtime.Serialization;

   6:  using System.Windows;

   7:  using System.Windows.Controls;

   8:  using System.Windows.Navigation;

   9:  using Microsoft.Phone.Controls;

  10:  using Microsoft.Phone.Shell;

  11:  using robchblogsamples.Resources;

12:

  13:  using Microsoft.WindowsAzure.MobileServices;

14:

  15:  namespace robchblogsamples

  16:  {

  17:      public class SiteToSearchItem

  18:      {

  19:          public int Id { get; set; }

20:

  21:          [DataMember(Name = "SiteName")]

  22:          public string SiteName { get; set; }

23:

  24:          [DataMember(Name = "Language")]

  25:          public string Language{ get; set; }

26:

  27:          [DataMember(Name = "Domain")]

  28:          public string Domain { get; set; }

29:

  30:          [DataMember(Name = "SearchUrlTemplate")]

  31:          public string SearchUrlTemplate { get; set; }

  32:      }

33:

  34:      public partial class MainPage : PhoneApplicationPage

  35:      {

  36:          private MobileServiceCollectionView items;

37:

  38:          private IMobileServiceTable _table = App.MobileService.GetTable();

39:

  40:          public MainPage()

  41:          {

  42:              InitializeComponent();

  43:          }

44:

  45:          private async void InsertItem(SiteToSearchItem item)

  46:          {

  47:              await _table.InsertAsync(item);

  48:              items.Add(item);

  49:          }

50:

  51:          private void RefreshItems()

  52:          {

  53:              items = _table

  54:                  .Where(item => item.Language == "en-us")

  55:                  .ToCollectionView();

  56:              ListItems.ItemsSource = items;

  57:          }

58:

  59:          private async void UpdateItem(SiteToSearchItem item)

  60:          {

  61:              await _table.UpdateAsync(item);

  62:              items.Remove(item);

  63:          }

64:

  65:          private void ButtonRefresh_Click(object sender, RoutedEventArgs e)

  66:          {

  67:              RefreshItems();

  68:          }

69:

  70:          private void ButtonSave_Click(object sender, RoutedEventArgs e)

  71:          {

  72:              var item = new SiteToSearchItem

  73:              {

  74:                  SiteName = _textboxSiteName.Text,

  75:                  Language = _textboxLanguage.Text,

  76:                  Domain = _textblockDomain.Text,

  77:                  SearchUrlTemplate = _textboxUrlTemplate.Text

  78:              };

79:

  80:              InsertItem(item);

  81:          }

82:

  83:          protected override void OnNavigatedTo(NavigationEventArgs e)

  84:          {

  85:              RefreshItems();

  86:          }

  87:      }

  88:  }

…. and this in MainPage.xaml:

   1:  <phone:PhoneApplicationPage

   2:      x:Class="robchblogsamples.MainPage"

   3:      xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

   4:      xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

   5:      xmlns:phone="clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone"

   6:      xmlns:shell="clr-namespace:Microsoft.Phone.Shell;assembly=Microsoft.Phone"

   7:      xmlns:d="http://schemas.microsoft.com/expression/blend/2008"

   8:      xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"

   9:      mc:Ignorable="d"

  10:      FontFamily="{StaticResource PhoneFontFamilyNormal}"

  11:      FontSize="{StaticResource PhoneFontSizeNormal}"

  12:      Foreground="{StaticResource PhoneForegroundBrush}"

  13:      SupportedOrientations="Portrait" Orientation="Portrait"

  14:      shell:SystemTray.IsVisible="True">

15:

16:

  17:      <Grid x:Name="LayoutRoot" Background="Transparent">

  18:          <Grid.RowDefinitions>

  19:              <RowDefinition Height="Auto"/>

  20:              <RowDefinition Height="*"/>

  21:          Grid.RowDefinitions>

22:

23:

  24:          <StackPanel x:Name="TitlePanel" Grid.Row="0" Margin="12,17,0,28">

  25:              <TextBlock Text="robchblogsamples" Style="{StaticResource PhoneTextNormalStyle}" Margin="12,0"/>

  26:          StackPanel>

  27:          <ScrollViewer Grid.Row="1" Margin="12,0,12,0">

  28:              <StackPanel x:Name="ContentPanel" Width="456">

  29:                  <TextBlock Text="Enter some text below and click Save to insert a new SiteToSearchItem into your database" TextWrapping="Wrap" Margin="12,0" Height="54"/>

  30:                  <StackPanel>

  31:                      <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">

  32:                          <TextBlock TextWrapping="Wrap" Text="Site Name:" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333"/>

  33:                          <TextBox x:Name="_textboxSiteName" Text="" Width="300" />

  34:                      StackPanel>

  35:                      <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">

  36:                          <TextBlock TextWrapping="Wrap" Text="Language:" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333"/>

  37:                          <TextBox x:Name="_textboxLanguage" Text="" Width="300" />

  38:                      StackPanel>

  39:                      <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">

  40:                          <TextBlock TextWrapping="Wrap" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333" Text="Domain:"/>

  41:                          <TextBox x:Name="_textblockDomain" Text="" Width="300" />

  42:                      StackPanel>

  43:                      <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">

  44:                          <TextBlock TextWrapping="Wrap" Text="Url Template:" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333"/>

  45:                          <TextBox x:Name="_textboxUrlTemplate" Text="" Width="300" />

  46:                      StackPanel>

  47:                  StackPanel>

  48:                  <Button x:Name="ButtonSave" Click="ButtonSave_Click" Content="Save" Margin="0"/>

  49:                  <TextBlock Text="Click refresh below to load the unfinished TodoItems from your database. Use the checkbox to complete and update your TodoItems" TextWrapping="Wrap" Margin="12,36,12,0" Height="107" />

  50:                  <Button x:Name="ButtonRefresh" Click="ButtonRefresh_Click" Content="Refresh" Height="72"/>

  51:                  <phone:LongListSelector x:Name="ListItems" Height="400">

  52:                      <phone:LongListSelector.ItemTemplate>

  53:                          <DataTemplate>

  54:                              <StackPanel>

  55:                                  <TextBlock TextWrapping="Wrap" Text="{Binding SiteName, FallbackValue=[SiteName]}" FontSize="32"/>

  56:                                  <TextBlock TextWrapping="Wrap" Text="{Binding Language, FallbackValue=[Language]}" FontSize="16"/>

  57:                                  <TextBlock TextWrapping="Wrap" Text="{Binding Domain, FallbackValue=[Domain]}" FontSize="16"/>

  58:                                  <TextBlock TextWrapping="Wrap" Text="{Binding SearchUrlTemplate, FallbackValue=[SearchUrlTemplate]}" FontSize="16"/>

  59:                              StackPanel>

  60:                          DataTemplate>

  61:                      phone:LongListSelector.ItemTemplate>

  62:                  phone:LongListSelector>

  63:              StackPanel>

64:

  65:          ScrollViewer>

66:

67:

  68:      Grid>

  69:  phone:PhoneApplicationPage>

Add some sites

Press F5, and add some sites. :-)

Remember … It’s a very simplistic “Admin UI”… For example, there is no delete. If you need to delete something, you can either:

Add that feature yourself, or
Use the Azure Portal to delete your SiteToSearchItem table, re-add the table, and then re-add the sites from our “Admin UI”

The “Other” part…

OK. Got that Azure part all done …

Back to our “Search On” application...

Now it’s time to pull the data down from Azure, in our new GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure method. Basically, this method delegates to another method, called GetItemsFromAzureMobileService, to download the list of “SiteToSearchItem” entries. Once we have our list of sites in a structured class, we’ll use LINQ to populate a List of strings for our phrase list, by adding all the Site Name’s (e.g. “Bing”) via the “SiteName” property from all the “SiteToSearchItem” items in the list of items from Azure.

   1:  private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(string language)

   2:  {

   3:      List itemsFromAzure =

   4:          GetItemsFromAzureMobileService(

   5:              string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),

   6:              _downloadTimeout);

7:

   8:      List<string> phrasesFromAzure = new List<string>(

   9:          from item in itemsFromAzure

  10:          select item.SiteName);

11:

  12:      return phrasesFromAzure;

  13:  }

Here’s what our SiteToSearchItem looks like:

   1:  public class SiteToSearchItem

   2:  {

   3:      public int Id { get; set; }

4:

   5:      public string Domain { get; set; }

   6:      public string SiteName { get; set; }

7:

   8:      public string SearchUrlTemplate { get; set; }

9:

  10:      public string Language { get; set; }

  11:  }

Now, since we have a data structure to keep our sites in, instead of a dictionary of strings, let’s use that for our built-in list of sites to search:

   1:  private List _sitesToSearchBuiltInList = new List()

   2:  {

   3:      { new SiteToSearchItem()

   4:          {   Language = "en-us",

   5:              SiteName = "Amazon",

   6:              Domain = "www.amazon.com",

   7:              SearchUrlTemplate = "http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" } },

8:

   9:      { new SiteToSearchItem()

  10:          {   Language = "en-us",

  11:              SiteName = "Bing",

  12:              Domain = "www.bing.com",

  13:              SearchUrlTemplate = "http://www.bing.com/?q={0}" } },

14:

  15:      { new SiteToSearchItem()

  16:          {   Language = "en-us",

  17:              SiteName = "CNN",

  18:              Domain = "www.CNN.com",

  19:              SearchUrlTemplate = "http://www.cnn.com/search/?query={0}" } },

20:

  21:      { new SiteToSearchItem()

  22:          {   Language = "en-us",

  23:              SiteName = "zzzz",

  24:              Domain = "www.zzzz.com",

  25:              SearchUrlTemplate = "zzzz" } },

26:

  27:      { new SiteToSearchItem()

  28:          {   Language = "en-us",

  29:              SiteName = "Dictionary.com",

  30:              Domain = "www.dictionary.com",

  31:              SearchUrlTemplate = "http://dictionary.reference.com/browse/{0}" } },

32:

  33:      { new SiteToSearchItem()

  34:          {   Language = "en-us",

  35:              SiteName = "Ebay",

  36:              Domain = "www.ebay.com",

  37:              SearchUrlTemplate = "http://www.ebay.com/sch/i.html?_nkw={0}" } },

38:

  39:      { new SiteToSearchItem()

  40:          {   Language = "en-us",

  41:              SiteName = "Facebook",

  42:              Domain = "www.facebook.com",

  43:              SearchUrlTemplate = "http://www.facebook.com/search/results.php?q={0}" } },

44:

  45:      { new SiteToSearchItem()

  46:          {   Language = "en-us",

  47:              SiteName = "Google",

  48:              Domain = "www.google.com",

  49:              SearchUrlTemplate = "https://www.google.com/search?hl=en&q={0}" } },

50:

  51:      { new SiteToSearchItem()

  52:          {   Language = "en-us",

  53:              SiteName = "Hulu",

  54:              Domain = "www.hulu.com",

  55:              SearchUrlTemplate = "http://www.hulu.com/#!search?q={0}" } },

56:

  57:      { new SiteToSearchItem()

  58:          {   Language = "en-us",

  59:              SiteName = "IMDB",

  60:              Domain = "www.imdb.com",

  61:              SearchUrlTemplate = "http://www.imdb.com/find?s=all&q={0}" } },

62:

  63:      { new SiteToSearchItem()

  64:          {   Language = "en-us",

  65:              SiteName = "Linked In",

  66:              Domain = "www.linkedin.com",

  67:              SearchUrlTemplate = "http://www.linkedin.com/search/fpsearch?keywords={0}" } },

68:

  69:      { new SiteToSearchItem()

  70:          {   Language = "en-us",

  71:              SiteName = "MSN",

  72:              Domain = "www.msn.com",

  73:              SearchUrlTemplate = "http://www.bing.com/search?scope=msn&q={0}" } },

74:

  75:      { new SiteToSearchItem()

  76:          {   Language = "en-us",

  77:              SiteName = "Twitter",

  78:              Domain = "www.twitter.com",

  79:              SearchUrlTemplate = "https://twitter.com/search/{0}" } },

80:

  81:      { new SiteToSearchItem()

  82:          {   Language = "en-us",

  83:              SiteName = "Weather.com",

  84:              Domain = "www.weather.com",

  85:              SearchUrlTemplate = "http://www.weather.com/info/sitesearch?q={0}" } },

86:

  87:      { new SiteToSearchItem()

  88:          {   Language = "en-us",

  89:              SiteName = "You Tube",

  90:              Domain = "www.youtube.com",

  91:              SearchUrlTemplate = "https://www.youtube.com/results?search_query={0}" } },

92:

  93:      { new SiteToSearchItem()

  94:          {   Language = "en-us",

  95:              SiteName = "Zillow",

  96:              Domain = "www.zillow.com",

  97:              SearchUrlTemplate = "http://www.zillow.com/homes/{0}/" } }

  98:  };

Similar to the GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure method, we’ll use the same LINQ technique to populate our list of phrases for the built-in sites, that we’ve just defined.

   1:  private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList()

   2:  {

   3:      return new List<string>(

   4:          from item in _sitesToSearchBuiltInList

   5:          select item.SiteName);

   6:  }

On to the GetItemsFromAzureMobileService method…

There are several ways we can do this. For example, we could use the Azure Mobile Services client library, just like the To Do List (which we turned into our fancy Admin UI app). Or … we could use the REST end points that expose all the data in our tables as an OData end point.

Let’s use the REST-ful approach. I picked this method, because you might not end up using Azure Mobile Services, and you can get more out of copying this code if you go another route. You could download your data from anywhere you can host a JSON list.

Just download the data from the endpoint and convert it to a List of SiteToSearchItem instances.

   1:  private List GetItemsFromAzureMobileService(string url, int downloadTimeout)

   2:  {

   3:      string json = DownloadString(url, downloadTimeout);

   4:      List itemsFromJson = FromJSON>(json != null ? json : "[]");

   5:      return itemsFromJson;

   6:  }

Now, let’s add the two new functions that’ll make that work, DownloadString, and FromJSON:

   1:  private string DownloadString(string uri, int timeoutInMilliseconds)

   2:  {

   3:      string result = null;

4:

   5:      ManualResetEvent webrequestCompleted = new ManualResetEvent(false);

   6:      AsyncCallback ac = new AsyncCallback((IAsyncResult iar) =>

   7:      {

   8:          try

   9:          {

  10:              HttpWebRequest request = (HttpWebRequest)iar.AsyncState;

  11:              HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(iar);

  12:              using (StreamReader sr = new StreamReader(response.GetResponseStream()))

  13:              {

  14:                  result = sr.ReadToEnd();

  15:              }

  16:              response.Close();

  17:          }

  18:          catch (Exception)

  19:          {

  20:          }

  21:          finally

  22:          {

  23:              webrequestCompleted.Set();

  24:          }

  25:      });

26:

  27:      HttpWebRequest webrequest = (HttpWebRequest)HttpWebRequest.Create(new Uri(uri));

  28:      IAsyncResult ar = webrequest.BeginGetResponse(ac, webrequest);

  29:      if (ar.CompletedSynchronously)

  30:      {

  31:          ac.Invoke(ar);

  32:      }

33:

  34:      if (webrequestCompleted.WaitOne(timeoutInMilliseconds))

  35:      {

  36:      }

37:

  38:      return result;

  39:  }

40:

  41:  private static T FromJSON(string json)

  42:  {

  43:      using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(json)))

  44:      {

  45:          DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));

  46:          return (T)serializer.ReadObject(ms);

  47:      }

  48:  }

Since we’re getting our data from the server, we also have to change how we find the URL template given a site name. Here’s how we’re going to do that now:

   1:  private string GetUrlTemplateForSiteName(string siteName, string language)

   2:  {

   3:      string url;

4:

   5:      if (null == (url = GetUrlTemplateForSiteNameFromBuiltInList(siteName, language)) &&

   6:          null == (url = GetUrlTemplateForSiteNameFromAzure(siteName, language)))

   7:      {

   8:          url = GetUrlTemplateForUnknownSite(siteName);

   9:      }

10:

  11:      return url;

  12:  }

13:

  14:  private string GetUrlTemplateForSiteNameFromBuiltInList(string siteName, string language)

  15:  {

  16:      return GetUrlTemplateForSiteNameFromList(siteName, language, _sitesToSearchBuiltInList);

  17:  }

18:

  19:  private string GetUrlTemplateForSiteNameFromAzure(string siteName, string language)

  20:  {

  21:      List itemsFromAzure =

  22:          GetItemsFromAzureMobileService(

  23:              string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),

  24:              _downloadTimeout);

25:

  26:      return GetUrlTemplateForSiteNameFromList(siteName, language, itemsFromAzure);

  27:  }

28:

  29:  private string GetUrlTemplateForSiteNameFromList(string siteName, string language, List items)

  30:  {

  31:      List matchingItems = new List(

  32:          from item in items

  33:          where item.SiteName.ToLower() == siteName.ToLower() && item.Language == language

  34:          select item);

35:

  36:      return matchingItems.Count > 0

  37:          ? GetUrlTemplateForSiteToSearch(matchingItems.First())

  38:          : null;

  39:  }

40:

  41:  private string GetUrlTemplateForSiteToSearch(SiteToSearchItem item)

  42:  {

  43:      return string.IsNullOrWhiteSpace(item.SearchUrlTemplate)

  44:          ? GetUrlTemplateForUnknownSite(item.Domain)

  45:          : item.SearchUrlTemplate;

  46:  }

47:

  48:  private string GetUrlTemplateForUnknownSite(string siteName)

  49:  {

  50:      return siteName.Length > 4 && siteName[siteName.Length - 4] != '.'

  51:          ? string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{0}")

  52:          : string.Format(_defaultUrlTemplateForUnknownSites, "site:" + siteName + "%20{0}");

  53:  }

As a performance optimization, since getting the URL template actually blocks on a network operation now (to fetch the data from Azure), let’s start that operation prior to letting the user know that we’re about to do the search. Like this (check out lines 8, 10, and 12):

   1:  private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      string siteName, findText;

4:

   5:      if (null != (siteName = await GetSiteName(queryString)) &&

   6:          null != (findText = await GetTextToFindOnSite(queryString, siteName)))

   7:      {

   8:          Task<string> asyncGetUrlTemplate = Task.Run<string>(() =>

                   { return GetUrlTemplateForSiteName(siteName, GetSpeechRecognitionLanguage()); });

9:

  10:          await Speak(string.Format("Searching {0} for {1}", siteName, findText));

11:

  12:          string siteUrlTemplate = await asyncGetUrlTemplate;

  13:          NavigateToUrl(string.Format(siteUrlTemplate, findText, siteName));

  14:      }

  15:  }

We’re almost ready to try it out … Almost. Just a few more changes…

Now, you might be wondering, who actually calls UpdatePhraseListsAsync? Great question. Copy this code into place, and you’ll know the answer:

   1:  private void EnsureInitVoiceCommandsOnBackgroundThread()

   2:  {

   3:      Task.Run(() => EnsureInitVoiceCommands());

   4:  }

5:

   6:  private async void EnsureInitVoiceCommands()

   7:  {

   8:      await VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));

   9:      await UpdatePhraseListsAsync();

  10:  }

11:

But, didn’t we already install the command sets from the vcd file in the MainPage constructor? Yep, we used to. But now, we’ll be doing that from the HandleNonVoiceCommandInvocation method for now.

   1:  private async void HandleNonVoiceCommandInvocation()

   2:  {

   3:      EnsureInitVoiceCommandsOnBackgroundThread();

4:

   5:      await Speak("What site would you like to search?");

   6:      SearchSiteVoiceCommand(new Dictionary<string, string>());

   7:  }

That’ll work, because, the first run of the app will go thru this method. Ideally, we’d really do something like this from a Background agent, PeriodTask like implementation, but, we’ll save that for another day.

Now we’re ready to try it out. Go for it. F5!!

Did it work? Put a breakpoint on the UpdatePhraseListsAsync method, and step thru…

Here’s the full listing for Search On…

MainPage.xaml.cs:

using System;

using System.Collections.Generic;

using System.Net;

using System.Windows;

using System.Windows.Navigation;

using Microsoft.Phone.Controls;

using Windows.Phone.Speech.VoiceCommands;

using Windows.Phone.Speech.Recognition;

using Windows.Phone.Speech.Synthesis;

using System.Threading.Tasks;

using Microsoft.Phone.Tasks;

using System.Text.RegularExpressions;

using System.Threading;

using System.Text;

using System.Linq;

using System.Runtime.Serialization.Json;

using System.IO;

namespace SearchOn

    public partial class MainPage : PhoneApplicationPage

        public MainPage()

            InitializeComponent();

        protected override void OnNavigatedTo(NavigationEventArgs e)

            base.OnNavigatedTo(e);

            if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))

                HandleVoiceCommand(NavigationContext.QueryString);

            else if (e.NavigationMode == NavigationMode.New)

                HandleNonVoiceCommandInvocation();

            else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)

                NavigationService.GoBack();

        private async void HandleNonVoiceCommandInvocation()

            EnsureInitVoiceCommandsOnBackgroundThread();

            await Speak("What site would you like to search?");

            SearchSiteVoiceCommand(new Dictionary<string, string>());

        private void HandleVoiceCommand(IDictionary<string, string> queryString)

            switch (queryString["voiceCommandName"])

                case "searchSite":

                case "searchSiteCatchAll":

                    SearchSiteVoiceCommand(queryString);

                    break;

        private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

            string siteName, findText;

            if (null != (siteName = await GetSiteName(queryString)) &&

                null != (findText = await GetTextToFindOnSite(queryString, siteName)))

                Task<string> asyncGetUrlTemplate = Task.Run<string>(() => { return GetUrlTemplateForSiteName(siteName, GetSpeechRecognitionLanguage()); });

                await Speak(string.Format("Searching {0} for {1}", siteName, findText));

                string siteUrlTemplate = await asyncGetUrlTemplate;

                NavigateToUrl(string.Format(siteUrlTemplate, findText, siteName));

        private async Task<string> GetSiteName(IDictionary<string, string> queryString)

            return queryString.ContainsKey("siteToSearch")

                ? queryString["siteToSearch"]

                : NormalizeDotComSuffixes(

                        await RecognizeTextFromWebSearchGrammar("Ex. \"msdn blogs\""));

        private async Task<string> GetTextToFindOnSite(IDictionary<string, string> queryString, string siteName)

            if (!queryString.ContainsKey("siteToSearch"))

                await Speak(string.Format("Search {0} for what?", siteName));

            return await RecognizeTextFromWebSearchGrammar("Ex. \"electronics\"");

        private async Task<string> RecognizeTextFromWebSearchGrammar(string exampleText)

            string text = null;

try

                SpeechRecognizerUI sr = new SpeechRecognizerUI();

                sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);

                sr.Settings.ListenText = "Listening...";

                sr.Settings.ExampleText = exampleText;

                sr.Settings.ReadoutEnabled = false;

                sr.Settings.ShowConfirmation = false;

                SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();

                if (result != null &&

                    result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&

                    result.RecognitionResult != null &&

                    result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)

                    text = result.RecognitionResult.Text;

            catch

            return text;

        private string GetSpeechRecognitionLanguage()

            return InstalledSpeechRecognizers.Default.Language.ToLower();

        private async Task Speak(string text)

            SpeechSynthesizer tts = new SpeechSynthesizer();

            await tts.SpeakTextAsync(text);

        private void NavigateToUrl(string url)

            WebBrowserTask task = new WebBrowserTask();

            task.Uri = new Uri(url, UriKind.Absolute);

            task.Show();

        private string GetUrlTemplateForSiteName(string siteName, string language)

            string url;

            if (null == (url = GetUrlTemplateForSiteNameFromBuiltInList(siteName, language)) &&

                null == (url = GetUrlTemplateForSiteNameFromAzure(siteName, language)))

                url = GetUrlTemplateForUnknownSite(siteName);

            return url;

        private string GetUrlTemplateForSiteNameFromBuiltInList(string siteName, string language)

            return GetUrlTemplateForSiteNameFromList(siteName, language, _sitesToSearchBuiltInList);

        private string GetUrlTemplateForSiteNameFromAzure(string siteName, string language)

            List itemsFromAzure =

                GetItemsFromAzureMobileService(

                    string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),

                    _downloadTimeout);

            return GetUrlTemplateForSiteNameFromList(siteName, language, itemsFromAzure);

        private string GetUrlTemplateForSiteNameFromList(string siteName, string language, List items)

            List matchingItems = new List(

                from item in items

                where item.SiteName.ToLower() == siteName.ToLower() && item.Language == language

                select item);

            return matchingItems.Count > 0

                ? GetUrlTemplateForSiteToSearch(matchingItems.First())

                : null;

        private string GetUrlTemplateForSiteToSearch(SiteToSearchItem item)

            return string.IsNullOrWhiteSpace(item.SearchUrlTemplate)

                ? GetUrlTemplateForUnknownSite(item.Domain)

                : item.SearchUrlTemplate;

        private string GetUrlTemplateForUnknownSite(string siteName)

            return siteName.Length > 4 && siteName[siteName.Length - 4] != '.'

                ? string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{0}")

                : string.Format(_defaultUrlTemplateForUnknownSites, "site:" + siteName + "%20{0}");

        private string NormalizeDotComSuffixes(string input)

            Regex re1 = new Regex(@"\s+dot\s+(?(com|net|org|gov))\z");

            if (input != null && re1.IsMatch(input))

                input = re1.Replace(input, @".${suffix}").Replace(" ", "");

            Regex re2 = new Regex(@"\s+dot\s+(?[A-Z])[.]\s+(?[A-Z])[.]\s+(?[A-Z])[.]\z");

            if (input != null && re2.IsMatch(input))

                input = re2.Replace(input, @".${l1}${l2}${l3}").Replace(" ", "");

                input = input.Substring(0, input.Length - 4) + input.Substring(input.Length - 4).ToLower();

            return input;

        private void EnsureInitVoiceCommandsOnBackgroundThread()

            Task.Run(() => EnsureInitVoiceCommands());

        private async void EnsureInitVoiceCommands()

            await VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));

            await UpdatePhraseListsAsync();

        private async Task UpdatePhraseListsAsync()

            foreach (VoiceCommandSet cs in VoiceCommandService.InstalledCommandSets.Values)

                List<string> updatedListOfPhrases = GetPhrasesForUpdatedSiteToSearchPhraseList(cs.Language.ToLower());

                await cs.UpdatePhraseListAsync("siteToSearch", updatedListOfPhrases);

        private List<string> GetPhrasesForUpdatedSiteToSearchPhraseList(string language)

            List<string> phrasesBuiltIn = GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList();

            List<string> phrasesFromAzure = GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(language);

            return phrasesFromAzure.Union(phrasesBuiltIn).ToList();

        private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList()

            return new List<string>(

                from item in _sitesToSearchBuiltInList

                select item.SiteName);

        private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(string language)

            List itemsFromAzure =

                GetItemsFromAzureMobileService(

                    string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),

                    _downloadTimeout);

            List<string> phrasesFromAzure = new List<string>(

                from item in itemsFromAzure

                select item.SiteName);

            return phrasesFromAzure;

        private List GetItemsFromAzureMobileService(string url, int downloadTimeout)

            string json = DownloadString(url, downloadTimeout);

            List itemsFromJson = FromJSON>(json != null ? json : "[]");

            return itemsFromJson;

        private string DownloadString(string uri, int timeoutInMilliseconds)

            string result = null;

            ManualResetEvent webrequestCompleted = new ManualResetEvent(false);

            AsyncCallback ac = new AsyncCallback((IAsyncResult iar) =>

try

                    HttpWebRequest request = (HttpWebRequest)iar.AsyncState;

                    HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(iar);

                    using (StreamReader sr = new StreamReader(response.GetResponseStream()))

                        result = sr.ReadToEnd();

                    response.Close();

                catch (Exception)

                finally

                    webrequestCompleted.Set();

});

            HttpWebRequest webrequest = (HttpWebRequest)HttpWebRequest.Create(new Uri(uri));

            IAsyncResult ar = webrequest.BeginGetResponse(ac, webrequest);

            if (ar.CompletedSynchronously)

                ac.Invoke(ar);

            if (webrequestCompleted.WaitOne(timeoutInMilliseconds))

            return result;

        private static T FromJSON(string json)

            using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(json)))

                DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));

                return (T)serializer.ReadObject(ms);

        #region private data

        private string _defaultUrlTemplateForUnknownSites = "http://m.bing.com/search?q={0}";

        private List _sitesToSearchBuiltInList = new List()

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Amazon",

                    Domain = "www.amazon.com",

                    SearchUrlTemplate = "http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Bing",

                    Domain = "www.bing.com",

                    SearchUrlTemplate = "http://www.bing.com/?q={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "CNN",

                    Domain = "www.CNN.com",

                    SearchUrlTemplate = "http://www.cnn.com/search/?query={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "zzzz",

                    Domain = "www.zzzz.com",

                    SearchUrlTemplate = "zzzz" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Dictionary.com",

                    Domain = "www.dictionary.com",

                    SearchUrlTemplate = "http://dictionary.reference.com/browse/{0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Ebay",

                    Domain = "www.ebay.com",

                    SearchUrlTemplate = "http://www.ebay.com/sch/i.html?_nkw={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Facebook",

                    Domain = "www.facebook.com",

                    SearchUrlTemplate = "http://www.facebook.com/search/results.php?q={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Google",

                    Domain = "www.google.com",

                    SearchUrlTemplate = "https://www.google.com/search?hl=en&q={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Hulu",

                    Domain = "www.hulu.com",

                    SearchUrlTemplate = "http://www.hulu.com/#!search?q={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "IMDB",

                    Domain = "www.imdb.com",

                    SearchUrlTemplate = "http://www.imdb.com/find?s=all&q={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Linked In",

                    Domain = "www.linkedin.com",

                    SearchUrlTemplate = "http://www.linkedin.com/search/fpsearch?keywords={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "MSN",

                    Domain = "www.msn.com",

                    SearchUrlTemplate = "http://www.bing.com/search?scope=msn&q={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Twitter",

                    Domain = "www.twitter.com",

                    SearchUrlTemplate = "https://twitter.com/search/{0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Weather.com",

                    Domain = "www.weather.com",

                    SearchUrlTemplate = "http://www.weather.com/info/sitesearch?q={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "You Tube",

                    Domain = "www.youtube.com",

                    SearchUrlTemplate = "https://www.youtube.com/results?search_query={0}" } },

            { new SiteToSearchItem()

                {   Language = "en-us",

                    SiteName = "Zillow",

                    Domain = "www.zillow.com",

                    SearchUrlTemplate = "http://www.zillow.com/homes/{0}/" } }

};

        private string _siteToSearchAzureMobileServicesTableUrlTemplate = "https://robchblogsamples.azure-mobile.net/tables/SiteToSearchItem?$filter=Language%20eq%20'{0}'";

        private int _downloadTimeout = 5000;

        #endregion

    public class SiteToSearchItem

        public int Id { get; set; }

        public string Domain { get; set; }

        public string SiteName { get; set; }

        public string SearchUrlTemplate { get; set; }

        public string Language { get; set; }

WP8 Voice Commands: Wildcards

Rob Chambers — Thu, 08 Nov 2012 15:44:58 GMT

In today’s post, we’re going to dig into a very useful feature of the Windows Phone 8.0 speech platform, specifically how to use wildcards with the VoiceCommandService.

As those of you following along already know, in our first post, we built a WP8 speech app, called “Search On”, enabling users to “Search On Amazon” by voice for anything they wanted. In the second post, we added the ability to search on 15+ more sites by incorporating a PhraseList of many popular web sites.

Today, we’ll continuing adding to our “Search On” speech app, enabling our users to search any web site at all, even when it’s not one of our built in 15+ sites we know how to search. Specifically, we’ll extend the app to support the following scenarios:

User: "Search a web site"
Phone: "What web site would you like to search?"

User: "Whitehouse Dot Gov"
Phone: "Search Whitehouse.gov for what?"

User: "Elections results"
Phone: "Searching Whitehouse.gov for Election results”

… and, similarly, if the user doesn’t know if we know about the site name or not …

User: "Search Whitehouse Dot Gov"
Phone: "What web site would you like to search?"

User: "Whitehouse Dot Gov"
Phone: "Search Whitehouse.gov for what?"

User: "Elections results"
Phone: "Searching Whitehouse.gov for Election results”

In the 2nd example above, it’s better to help the user complete her task, than just beeping at her, or complaining that we don’t know what to do at all. We DO know at least part of the task: She wants to search a web site, but we just didn’t know what the site was she said…

Now, let’s dig in and see how WP8’s VoiceCommandService can help us achieve this…

Let’s start with the Voice Command Definition file, vcd.xml. We’re going to add a new Command, named “searchSiteCatchAll”. We’ll add a few elements for that Command, that will “capture” all the websites that we don’t know how to search yet.

   1:  <Command Name="searchSiteCatchAll">

   2:    <Example>a web siteExample>

   3:    <ListenFor>[a] [specific] [web] siteListenFor>

   4:    <ListenFor>{*} ListenFor>

   5:    <ListenFor>{*} {dotComOrNet} ListenFor>

   6:    <Feedback>What site would you like to search?Feedback>

   7:    <Navigate Target="MainPage.xaml" />

   8:  Command>

Remember how we used a PhraseList in yesterday’s post to contain a list of websites? Then, we added “{siteToSearch}" as the content to the element in the element? Right? Well, the VoiceCommandService wildcard feature, or garbage rule as it’s sometimes called, is very similar. Instead of referring to a PhraseList by Label, just use an asterisks inside the same squiggly brackets we put around the PhraseList Label. So it’s “{*}” instead of “{siteToSearch}”.

Now, you can see that I also added a that doesn’t use the wildcard. That will allow the user to say things like, “Search On a website”, or “Search On a specific site”. That way, if the user already knows we don’t know about the site, she can get into the same flow, more deterministically.

OK, now you’ll see that there’s another new PhraseList being referred to here. Named “dotComOrNet”. Let’s add that PhraseList now:

   1:  <PhraseList Label="dotComOrNet" Disambiguate="false">

   2:    <Item>dot comItem>

   3:    <Item>dot netItem>

   4:    <Item>dot orgItem>

   5:    <Item>dot govItem>

   6:  PhraseList>

We use that as the “tail” of the ListenFor to help the recognizer on WP8 to choose our application instead of the built in search experience in WP8. Without this “tail”, due to the fact that our application name is phonetically very similar to a build in command’s verb “Search”, the 1st party experience will get picked more often than we’d expect. I’ll save describing why that is for another day, though (for the very curious it’s because of some internal weighting we’ve done to ensure the built in WP8 speech features can more reliably be accessed, even if/when poorly designed 3rd party applications might be on the phone with phonetically similar voice command phrases).

Also, did you notice the “Disambiguate=”false”” part? What’s that do? Good question … Basically, it tells the VoiceCommandService to not worry about asking the user to pick between the different choices if there’s ever any ambiguity. We don’t really need to know what they said, for our built in items. We’ll take a look at this more closely in a future post.

OK. Now, I’ve also updated the “siteToSearch” command, to also take advantage of this “tail”. Combined with the catch all command, it looks like this:

   1:  <Command Name="searchSiteCatchAll">

   2:    <Example>a web siteExample>

   3:    <ListenFor>[a] [specific] [web] siteListenFor>

   4:    <ListenFor>{*} ListenFor>

   5:    <ListenFor>{*} {dotComOrNet} ListenFor>

   6:    <Feedback>What site would you like to search?Feedback>

   7:    <Navigate Target="MainPage.xaml" />

   8:  Command>

9:

  10:  <Command Name="searchSite">

  11:    <Example>Amazon, Bing, Facebook, Twitter, ...Example>

  12:    <ListenFor>{siteToSearch} ListenFor>

  13:    <ListenFor>{siteToSearch} {dotComOrNet}ListenFor>

  14:    <Feedback>Search on {siteToSearch} for what?Feedback>

  15:    <Navigate Target="MainPage.xaml" />

  16:  Command>

On to the code!

First, let’s look at the HandleVoiceCommand function.

   1:  private void HandleVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      switch (queryString["voiceCommandName"])

   4:      {

   5:          case "searchSite":

   6:          case "searchSiteCatchAll":

   7:              SearchSiteVoiceCommand(queryString);

   8:              break;

   9:      }

  10:  }

Now it’s set up to handle more than one voice command, but for now, it’s going to do the same thing for both searchSite and searchSiteCatchAll commands. For that to work, though, we’ll have to also change the SearchSiteVoiceCommand method to look like this:

   1:  private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      string siteName, findText;

4:

   5:      if (null != (siteName = await GetSiteName(queryString)) &&

   6:          null != (findText = await GetTextToFindOnSite(queryString, siteName)))

   7:      {

   8:          await Speak(string.Format("Searching {0} for {1}", siteName, findText));

9:

  10:          string siteUrlTemplate = GetUrlTemplateFromSiteName(siteName);

  11:          NavigateToUrl(string.Format(siteUrlTemplate, findText, siteName));

  12:      }

  13:  }

Basically, we’ll get the site name, from a new method called GetSiteName, and we’ll get the text to find from a new method as well, called GetTextToFindOnSite. Let’s take a look at both of those methods:

   1:  private async Task<string> GetSiteName(IDictionary<string, string> queryString)

   2:  {

   3:      return queryString.ContainsKey("siteToSearch")

   4:          ? queryString["siteToSearch"]

   5:          : NormalizeDotComSuffixes(

   6:                  await RecognizeTextFromWebSearchGrammar("Ex. \"msdn blogs\""));

   7:  }

8:

   9:  private async Task<string> GetTextToFindOnSite(IDictionary<string, string> queryString, string siteName)

  10:  {

  11:      if (!queryString.ContainsKey("siteToSearch"))

  12:      {

  13:          await Speak(string.Format("Search {0} for what?", siteName));

  14:      }

15:

  16:      return await RecognizeTextFromWebSearchGrammar("Ex. \"electronics\"");

  17:  }

In the first method, GetSiteName, we’ll check the dictionary, and if it has the siteToSearch, we’ll use that. But, if it doesn’t, we’ll get that site name from the same method we were getting the text to search for before, RecognizeTextFromWebSearchGrammar. Previously, though, that method had a hard coded example string, but now that we’re using it in two different ways, we’ll pass the example text string in as a parameter.

The second new method, GetTextToFindOnSite, similarly calls off to RecognizeTextFromWebSearchGrammar, but it also optionally prompts the user. This is because in the case of the “searchSite” voice command, the prompt will have already been played, by the VoiceCommandService’s use of that command’s element. However, for “searchSiteCatchAll”, the prompt from the VoiceCommandService is asking the user to will be asking what site to search. So, once the user responds to that, we’ll need to prompt them for the text they’d like to search for. Clear as mud? Well, hopefully it’s at least as clear as chicken broth. :-)

The actual implementation of RecognizeTextFromWebSearchGrammar is almost identical; only two lines are different: line 1 has a new parameter being passed in, the exampleText, and line 9, where we use that instead of the hard coded example from before.

   1:  private async Task<string> RecognizeTextFromWebSearchGrammar(string exampleText)

   2:  {

   3:      string text = null;

   4:      try

   5:      {

   6:          SpeechRecognizerUI sr = new SpeechRecognizerUI();

   7:          sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);

   8:          sr.Settings.ListenText = "Listening...";

   9:          sr.Settings.ExampleText = exampleText;

  10:          sr.Settings.ReadoutEnabled = false;

  11:          sr.Settings.ShowConfirmation = false;

12:

  13:          SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();

  14:          if (result != null &&

  15:              result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&

  16:              result.RecognitionResult != null &&

  17:              result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)

  18:          {

  19:              text = result.RecognitionResult.Text;

  20:          }

  21:      }

  22:      catch

  23:      {

  24:      }

  25:      return text;

  26:  }

OK… The next change we’re going to make is to the GetUrlTemplateFromSiteName method. Previously, this was just looking up the site name in the dictionary of strings for our 15+ sites, and if it couldn’t find it, it would use a default URL template. Very similar to that still, now it also uses the “site:____.__” feature of Bing search, as long as it thinks that the siteName is actually a web site. I’m checking that in a fairly lame fashion for this example, but it works pretty well. Feel free to do a better check here yourself (see lines 9-12). :-)

   1:  private string GetUrlTemplateFromSiteName(string siteName)

   2:  {

   3:      string url = null;

4:

   5:      if (_siteUrlTemplateDictionary.ContainsKey(siteName))

   6:      {

   7:          url = _siteUrlTemplateDictionary[siteName];

   8:      }

   9:      else if (siteName.Length > 4 && siteName[siteName.Length - 4] != '.')

  10:      {

  11:          url = string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{0}");

  12:      }

  13:      else

  14:      {

  15:          url = string.Format(_defaultUrlTemplateForUnknownSites, "site:" + siteName + "%20{0}");

  16:      }

17:

  18:      return url;

  19:  }

Last substantive change coming up … In our new GetSiteName method up above, I snuck in a call to a 3rd new method, NormalizeDotComSuffixes. It, well … Normalizes the site name we get back from voice search by trying to convert the tail of the string from things like “ dot com” and “dot O. R. G.” into “.com” and “.org”. That’ll be pretty important to make the new “site:____.___” Bing site specific search feature work in our app. Don’t forget to add a “using System.Text.RegularExpressions;” up at the top as well.

   1:  private string NormalizeDotComSuffixes(string input)

   2:  {

   3:      Regex re1 = new Regex(@"\s+dot\s+(?(com|net|org|gov))\z");

   4:      if (input != null && re1.IsMatch(input))

   5:      {

   6:          input = re1.Replace(input, @".${suffix}").Replace(" ", "");

   7:      }

8:

   9:      Regex re2 = new Regex(@"\s+dot\s+(?[A-Z])[.]\s+(?[A-Z])[.]\s+(?[A-Z])[.]\z");

  10:      if (input != null && re2.IsMatch(input))

  11:      {

  12:          input = re2.Replace(input, @".${l1}${l2}${l3}").Replace(" ", "");

  13:          input = input.Substring(0, input.Length - 4) + input.Substring(input.Length - 4).ToLower();

  14:      }

15:

  16:      return input;

  17:  }

OK … Now, here’s the final change before we can test out our 3rd version of “Search On” …

Remember when the app was launched from the Start menu, or in any other non-VoiceCommandService type of way yesterday? When that happened, we did a search over to my blog. That’s neat and all, but … Now that we have the ability to ask the user what site they want to search, let’s just go ahead and start there in the conversation if our app is launched in t non-VoiceCommand kind of way.

To do that, we’ll make a super simple change to OnNavigatedTo (see line11, and a new function, called HandleNonVoiceCommandInvocation):

   1:  protected override void OnNavigatedTo(NavigationEventArgs e)

   2:  {

   3:      base.OnNavigatedTo(e);

4:

   5:      if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))

   6:      {

   7:          HandleVoiceCommand(NavigationContext.QueryString);

   8:      }

   9:      else if (e.NavigationMode == NavigationMode.New)

  10:      {

  11:          HandleNonVoiceCommandInvocation();

  12:      }

  13:      else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)

  14:      {

  15:          NavigationService.GoBack();

  16:      }

  17:  }

18:

  19:  private async void HandleNonVoiceCommandInvocation()

  20:  {

  21:      await Speak("What site would you like to search?");

  22:      SearchSiteVoiceCommand(new Dictionary<string, string>());

  23:  }

Basically, we’ll just prompt the user, and then do a site search just like we do when we get a “searchSiteCatchAll” voice command. That’s it.

OK. Time to give it a run! Press F5 and let’s take it for a spin.

You: "Whitehouse Dot Gov"
Phone: "Search Whitehouse.gov for what?"

You: "Elections results"
Phone: "Searching Whitehouse.gov for Election results”

Did it work? It did for me. Yeah!

Well, what about the voice command path into the code? Let’s try that out too … Press and hold the “Start” button and follow along below…

User: "Search a web site"
Phone: "What web site would you like to search?"

User: "Whitehouse Dot Gov"
Phone: "Search Whitehouse.gov for what?"

User: "Elections results"
Phone: "Searching Whitehouse.gov for Election results”

… and, similarly …

User: "Search Whitehouse Dot Gov"
Phone: "What web site would you like to search?"

User: "Whitehouse Dot Gov"
Phone: "Search Whitehouse.gov for what?"

User: "Elections results"
Phone: "Searching Whitehouse.gov for Election results”

Did those both work too?

OK. So that’s it for today… What’d we learn? Hopefully, you learned a few things:

You can use {*} very similarly to how you use {phraseListLabel}, without defining the list
You can have more than one element in the
You can use the “Disambiguate” attribute on your elements to tell the VoiceCommandService to not worry about clarifying which one the user said, in cases of ambiguity.
And … Last by not least … You’ve learned that I really enjoy typing three dots in a row in all my posts … Like that … And like this … :-)

Keep scrolling down for the full listings for our two most important source files. Catch you later!

vcd.xml

xml version="1.0" encoding="utf-8"?>

<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0">

  <CommandSet xml:lang="en-US">

    <CommandPrefix>Search OnCommandPrefix>

    <Example>Amazon, Bing, Facebook, Twitter, ...Example>

    <Command Name="searchSiteCatchAll">

      <Example>a web siteExample>

      <ListenFor>[a] [specific] [web] siteListenFor>

      <ListenFor>{*} ListenFor>

      <ListenFor>{*} {dotComOrNet} ListenFor>

      <Feedback>What site would you like to search?Feedback>

      <Navigate Target="MainPage.xaml" />

    Command>

    <Command Name="searchSite">

      <Example>Amazon, Bing, Facebook, Twitter, ...Example>

      <ListenFor>{siteToSearch} ListenFor>

      <ListenFor>{siteToSearch} {dotComOrNet}ListenFor>

      <Feedback>Search on {siteToSearch} for what?Feedback>

      <Navigate Target="MainPage.xaml" />

    Command>

    <PhraseList Label="siteToSearch">

      <Item>AmazonItem>

      <Item>BingItem>

      <Item>CNNItem>

      <Item>Dictionary.comItem>

      <Item>EbayItem>

      <Item>FacebookItem>

      <Item>GoogleItem>

      <Item>HuluItem>

      <Item>IMDBItem>

      <Item>KayakItem>

      <Item>Linked InItem>

      <Item>MSNItem>

      <Item>NetflixItem>

      <Item>TwitterItem>

      <Item>Weather.comItem>

      <Item>YouTubeItem>

      <Item>ZillowItem>

    PhraseList>

    <PhraseList Label="dotComOrNet" Disambiguate="false">

      <Item>dot comItem>

      <Item>dot netItem>

      <Item>dot orgItem>

      <Item>dot govItem>

    PhraseList>

  CommandSet>

VoiceCommands>

MainPage.xaml.cs

using System;

using System.Collections.Generic;

using System.Net;

using System.Windows;

using System.Windows.Navigation;

using Microsoft.Phone.Controls;

using Windows.Phone.Speech.VoiceCommands;

using Windows.Phone.Speech.Recognition;

using Windows.Phone.Speech.Synthesis;

using System.Threading.Tasks;

using Microsoft.Phone.Tasks;

using System.Text.RegularExpressions;

namespace SearchOn

    public partial class MainPage : PhoneApplicationPage

        public MainPage()

            InitializeComponent();

            VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));

        protected override void OnNavigatedTo(NavigationEventArgs e)

            base.OnNavigatedTo(e);

            if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))

                HandleVoiceCommand(NavigationContext.QueryString);

            else if (e.NavigationMode == NavigationMode.New)

                HandleNonVoiceCommandInvocation();

            else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)

                NavigationService.GoBack();

        private async void HandleNonVoiceCommandInvocation()

            await Speak("What site would you like to search?");

            SearchSiteVoiceCommand(new Dictionary<string, string>());

        private void HandleVoiceCommand(IDictionary<string, string> queryString)

            switch (queryString["voiceCommandName"])

                case "searchSite":

                case "searchSiteCatchAll":

                    SearchSiteVoiceCommand(queryString);

                    break;

        private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

            string siteName, findText;

            if (null != (siteName = await GetSiteName(queryString)) &&

                null != (findText = await GetTextToFindOnSite(queryString, siteName)))

                await Speak(string.Format("Searching {0} for {1}", siteName, findText));

                string siteUrlTemplate = GetUrlTemplateFromSiteName(siteName);

                NavigateToUrl(string.Format(siteUrlTemplate, findText, siteName));

        private async Task<string> GetSiteName(IDictionary<string, string> queryString)

            return queryString.ContainsKey("siteToSearch")

                ? queryString["siteToSearch"]

                : NormalizeDotComSuffixes(

                        await RecognizeTextFromWebSearchGrammar("Ex. \"msdn blogs\""));

        private async Task<string> GetTextToFindOnSite(IDictionary<string, string> queryString, string siteName)

            if (!queryString.ContainsKey("siteToSearch"))

                await Speak(string.Format("Search {0} for what?", siteName));

            return await RecognizeTextFromWebSearchGrammar("Ex. \"electronics\"");

        private async Task<string> RecognizeTextFromWebSearchGrammar(string exampleText)

            string text = null;

try

                SpeechRecognizerUI sr = new SpeechRecognizerUI();

                sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);

                sr.Settings.ListenText = "Listening...";

                sr.Settings.ExampleText = exampleText;

                sr.Settings.ReadoutEnabled = false;

                sr.Settings.ShowConfirmation = false;

                SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();

                if (result != null &&

                    result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&

                    result.RecognitionResult != null &&

                    result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)

                    text = result.RecognitionResult.Text;

            catch

            return text;

        private async Task Speak(string text)

            SpeechSynthesizer tts = new SpeechSynthesizer();

            await tts.SpeakTextAsync(text);

        private void NavigateToUrl(string url)

            WebBrowserTask task = new WebBrowserTask();

            task.Uri = new Uri(url, UriKind.Absolute);

            task.Show();

        private string GetUrlTemplateFromSiteName(string siteName)

            string url = null;

            if (_siteUrlTemplateDictionary.ContainsKey(siteName))

                url = _siteUrlTemplateDictionary[siteName];

            else if (siteName.Length > 4 && siteName[siteName.Length - 4] != '.')

                url = string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{0}");

            else

                url = string.Format(_defaultUrlTemplateForUnknownSites, "site:" + siteName + "%20{0}");

            return url;

        private string NormalizeDotComSuffixes(string input)

            Regex re1 = new Regex(@"\s+dot\s+(?(com|net|org|gov))\z");

            if (input != null && re1.IsMatch(input))

                input = re1.Replace(input, @".${suffix}").Replace(" ", "");

            Regex re2 = new Regex(@"\s+dot\s+(?[A-Z])[.]\s+(?[A-Z])[.]\s+(?[A-Z])[.]\z");

            if (input != null && re2.IsMatch(input))

                input = re2.Replace(input, @".${l1}${l2}${l3}").Replace(" ", "");

                input = input.Substring(0, input.Length - 4) + input.Substring(input.Length - 4).ToLower();

            return input;

        private string _defaultUrlTemplateForUnknownSites = "http://m.bing.com/search?q={0}";

        private Dictionary<string, string> _siteUrlTemplateDictionary = new Dictionary<string, string>()

            { "Amazon", "http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" },

            { "Bing", "http://www.bing.com/?q={0}" },

            { "CNN", "http://www.cnn.com/search/?query={0}" },

            { "Dictionary.com", "http://dictionary.reference.com/browse/{0}" },

            { "Ebay", "http://www.ebay.com/sch/i.html?_nkw={0}" },

            { "Facebook", "http://www.facebook.com/search/results.php?q={0}" },

            { "Google", "https://www.google.com/search?hl=en&q={0}" },

            { "Hulu", "http://www.hulu.com/#!search?q={0}" },

            { "IMDB", "http://www.imdb.com/find?s=all&q={0}" },

            { "Linked In", "http://www.linkedin.com/search/fpsearch?keywords={0}" },

            { "MSN", "http://www.bing.com/search?scope=msn&q={0}" },

            { "Twitter", "https://twitter.com/search/{0}" },

            { "Weather", "http://www.weather.com/info/sitesearch?q={0}" },

            { "YouTube", "https://www.youtube.com/results?search_query={0}" },

            { "Zillow", "http://www.zillow.com/homes/{0}/" },

};

WP8 Voice Commands: Phrase Lists

Rob Chambers — Wed, 07 Nov 2012 12:00:14 GMT

Today, I ’m going to show you how to extend the app we built in yesterday’s post, using something we call Phrase Lists. Using a Phrase List, we’ll be able to let users of our “Search On” application “Search on” 15 more sites, not just search on Amazon.com.

Here are a couple examples of how that’ll play out for users of the app…

To search on Amazon, it’ll work just like it did yesterday (with a small tweak to the prompt):

User: "Search On Amazon"
Phone: "Search Amazon for what?"
User: "Electronics" (or whatever the user wants)
Phone: "Searching Amazon for Electronics" (or whatever the user said she wanted)

But with the changes today, you’ll also be able to search on a bunch of sites. Here are the sites we’ll add today: Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook, MSN, Twitter, Weather.com, YouTube, or Zillow. Feel free to add more when you’re typing it up on your computer.

User: "Search On CNN"
Phone: "Search CNN for what?"
User: "Presidential Election Results”
Phone: "Searching CNN for Presidential Election Results"

… or …

User: "Search On Weather.com"
Phone: "Search Weather.com for what?"
User: "Seattle 10 day forecast”
Phone: "Searching Weather.com for Seattle 10 day forecast"

Sound Good? OK… Let’s get started.

First of all, if you haven’t built the app from yesterday, go do that now. Start here…

Now, let’s update our Voice Command Definition file. We’re going to make 4 changes to the VCD file. 1.) We’re going to update the examples phrases that get displayed to users in the built in speech UX in WP8, 2.) We’re going to add a Phrase List that will contain the list of sites that we want to be able to search, 3.) We’ll refer to that PhraseList from our searchSite , specifically, from our elements, and, finally 4.) We’ll change the text feedback in the element to react dynamically to what the user said.

Here we go …

To update the example phrases, simply find the existing elements from the vcd.xml file, and replace them this:

<Example>Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook,

         MSN, Twitter, Weather.com, YouTube, or ZillowExample>

Now, let’s actually define the PhraseList. You can have up to 10 PhraseLists in your VoiceCommands CommandSet, and the PhraseLists must be after all the Command elements in the XML file. If there are more than 10, or they’re not in the right spot in the VCD file, Visual Studio should give you a red underlined squiggly line that will tell you what’s wrong when you hover over it. If for some reason you miss this, at runtime the call to InstallCommandSetsFromFileAsync will throw an exception. In our sample, we’re not trying to catch those, so if you don’t have a debugger hooked up, you’ll miss it.

We’ll name our PhraseList “siteToSearch”, and define it like this:

   1:  <PhraseList Label="siteToSearch">

   2:    <Item>AmazonItem>

   3:    <Item>BingItem>

   4:    <Item>CNNItem>

   5:    <Item>Dictionary.comItem>

   6:    <Item>EbayItem>

   7:    <Item>FacebookItem>

   8:    <Item>GoogleItem>

   9:    <Item>HuluItem>

  10:    <Item>IMDBItem>

  11:    <Item>KayakItem>

  12:    <Item>Linked InItem>

  13:    <Item>MSNItem>

  14:    <Item>NetflixItem>

  15:    <Item>TwitterItem>

  16:    <Item>Weather.comItem>

  17:    <Item>YouTubeItem>

  18:    <Item>ZillowItem>

  19:  PhraseList>

Next, let’s update our Command, with ListenFor elements that reference this list.

The text inside a ListenFor element is used as a constraint for the recognizer. In yesterday’s example, we used a simple string, “Amazon”, as the content for our one and only ListenFor element. That meant that the user had to say “Search On”, which is the command prefix for our application, followed by “Amazon”. If you have several things you want the user to be able to say, you can do one, or both, of the following 2 things: have more than one element in your command, and/or have your element refer to a PhraseList by it’s Label.

Let’s see what that looks like for our updated application. Replace the from yesterday, with the following:

   1:  <ListenFor>{siteToSearch}ListenFor>

   2:  <ListenFor>{siteToSearch} dot comListenFor>

The first element only contains a reference to the PhraseList. The second , though, both references the same PhraseList, but it also has the words “dot com” after that PhraseList reference. What does this do? Well, it tells the recognizer that it’s OK for the user to say any one of the items inside the PhraseList with the siteToSearch Label, followed by the phrase “dot com”. Because we have both of these elements, it essentially makes the “dot com” and optional phrase. We could have specified using an “optional” notation, using square brackets, like this:

   1:  <ListenFor>{siteToSearch} [dot com]ListenFor>

Either of these two representations is essentially result in the same constraint being communicated to the speech recognizer.

OK. We’ve updated our examples, we’ve added our PhraseList for the sites to search, and we’ve updated our phrases for our command. Last thing we need to do is to update the element.

Similar to the elements’ use of the PhraseList Label, we can substitute what was said from the PhraseList into the text feedback as well. Let’s update our element, and it should be pretty obvious what I mean:

   1:  <Feedback>Search on {siteToSearch} for what?Feedback>

What that’ll do is this: Once the VoiceCommandService determines that one of your commands was recognized, it’ll take the text inside the corresponding element, and it’ll replace all occurrences of curly brace delimited PhraseLists with the text for the item that was actually recognized from the user speaking. That way, if I said “Search On Amazon”, the VoiceCommandService will replace the “{siteToSearch}” part of “Search on {siteToSearch} for what?” feedback string with “Amazon”. It’s a lot like C# string formatting, except using PhraseList Labels instead of order based parameter references, like 0, 1, 2.

OK. Now, our VCD file is all up to date. Great. Now let’s turn our attention to the code in MainPage.xaml.cs.

Let’s start off by adding a Dictionary of strings that we’ll use to look up the URL for the site we’ll be searching. And, if let’s have a backup way of searching if we can’t find that URL. Scroll down to the bottom of the MainPage class, and add the following:

   1:  private string _defaultUrlTemplateForUnknownSites = "http://m.bing.com/search?q={0}";

2:

   3:  private Dictionary<string, string> _siteUrlTemplateDictionary = new Dictionary<string, string>()

   4:  {

   5:      { "Amazon", "http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" },

   6:      { "Bing", "http://www.bing.com/?q={0}" },

   7:      { "CNN", "http://www.cnn.com/search/?query={0}" },

   8:      { "Dictionary.com", "http://dictionary.reference.com/browse/{0}" },

   9:      { "Ebay", "http://www.ebay.com/sch/i.html?_nkw={0}" },

  10:      { "Facebook", "http://www.facebook.com/search/results.php?q={0}" },

  11:      { "Google", "https://www.google.com/search?hl=en&q={0}" },

  12:      { "Hulu", "http://www.hulu.com/#!search?q={0}" },

  13:      { "IMDB", "http://www.imdb.com/find?s=all&q={0}" },

  14:      { "Linked In", "http://www.linkedin.com/search/fpsearch?keywords={0}" },

  15:      { "MSN", "http://www.bing.com/search?scope=msn&q={0}" },

  16:      { "Twitter", "https://twitter.com/search/{0}" },

  17:      { "Weather", "http://www.weather.com/info/sitesearch?q={0}" },

  18:      { "YouTube", "https://www.youtube.com/results?search_query={0}" },

  19:      { "Zillow", "http://www.zillow.com/homes/{0}/" },

  20:  };

Now, let’s add a method that’ll do the lookup for us:

   1:  private string GetUrlTemplateFromSiteName(string siteName)

   2:  {

   3:      return _siteUrlTemplateDictionary.ContainsKey(siteName)

   4:          ? _siteUrlTemplateDictionary[siteName]

   5:          : string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{1}");

   6:  }

Great. Now let’s update our SearchSiteVoiceCommand method to figure out what site we’re actually searching, update how we speak the feedback based on the site name, use our new GetUrlTemplateFromSiteName method to find the right URL template, and then stick the search string into the URL template appropriately. When it’s all done, it should look like this:

   1:  private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      string text = await RecognizeTextFromWebSearchGrammar();

   4:      if (text != null)

   5:      {

   6:          string siteName = queryString["siteToSearch"];

   7:          await Speak(string.Format("Searching {0} for {1}", siteName, text));

8:

   9:          string siteUrlTemplate = GetUrlTemplateFromSiteName(siteName);

  10:          NavigateToUrl(string.Format(siteUrlTemplate, text, siteName));

  11:      }

  12:  }

One additional thing I’ll do, that we don’t absolutely need to at this point, but I think it’s good practice, is to make sure that we actually have a “siteToSearch” item in our query string dictionary in our HandleVoiceCommand method.

   1:  private void HandleVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      if (queryString["voiceCommandName"] == "searchSite" &&

   4:          queryString.ContainsKey("siteToSearch"))

   5:      {

   6:          SearchSiteVoiceCommand(queryString);

   7:      }

   8:  }

One more change, and then we can test it out.

Since we were only searching Amazon.com yesterday, it was probably OK for us to open up the Amazon.com home page if you started the Search On application from the Start menu, not by voice. But … It doesn’t make as much sense now, since we can search 15 different sites. Right?

So, I’ll update the code to search for blog posts by me about Windows Phone 8 on my MSDN blog instead. A bit self serving, I know, but … Hey … It’s my sample app, right? :-)

First, I’ll define a class variable to hold the new location we’ll open in that situation:

   1:  private string _defaultUrlForNewNavigations =

   2:      "http://m.bing.com/search?q=Rob's%20Rhapsody%20Windows%20Phone%208%20site:blogs.msdn.com";

And then, I’ll update the OnNavigatedTo method to use that new class variable (see line 11 in the snippet below):

   1:  protected override void OnNavigatedTo(NavigationEventArgs e)

   2:  {

   3:      base.OnNavigatedTo(e);

4:

   5:      if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))

   6:      {

   7:          HandleVoiceCommand(NavigationContext.QueryString);

   8:      }

   9:      else if (e.NavigationMode == NavigationMode.New)

  10:      {

  11:          NavigateToUrl(_defaultUrlForNewNavigations);

  12:      }

  13:      else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)

  14:      {

  15:          NavigationService.GoBack();

  16:      }

  17:  }

Now we’re ready to try it out.

If you haven’t typed this in all along the way, you can scroll down below and copy/paste into Visual Studio and give it a go that way.

Press F5, and you should see a Bing Search for my blog. Yeah.

Now, on the phone, Press and hold the “Start” menu, and let’s try out our examples from above.

You: "Search On Amazon"
Phone: "Search Amazon for what?"
You: "Electronics" (or whatever you want)
Phone: "Searching Amazon for Electronics" (or whatever you said you wanted)

Great. That one still works! Let’s try out the others…

User: "Search On CNN"
Phone: "Search CNN for what?"
User: "Presidential Election Results”
Phone: "Searching CNN for Presidential Election Results"

Did that one work as well? Mine did. :-) Last one … we’ll try here today…

User: "Search On Weather.com"
Phone: "Search Weather.com for what?"
User: "Seattle 10 day forecast”
Phone: "Searching Weather.com for Seattle 10 day forecast"

Excellent. Well … except it’s going to rain this week. Bummer. But the speech part worked. If only I could say “Clouds, stop raining!!” I’m sure I can do the speech part of that one; it’s the actual stop raining part I don’t know how to do. Yet. :-)

Well. That’s it for today. Hopefully, you now know how to use PhraseLists with the VoiceCommandService. And … Our “Search On” application is starting to actually become useful. I wonder how far we can take this …

Drop me some feedback below and let me know what you’d like to see, and I’ll try to work that into posts in the future.

Happy coding! … Oh yeah … I almost forgot … Here’s the full listing for the code, and the VCD file.

MainPage.xaml.cs

   1:  using System;

   2:  using System.Collections.Generic;

   3:  using System.Net;

   4:  using System.Windows;

   5:  using System.Windows.Navigation;

   6:  using Microsoft.Phone.Controls;

   7:  using Windows.Phone.Speech.VoiceCommands;

   8:  using Windows.Phone.Speech.Recognition;

   9:  using Windows.Phone.Speech.Synthesis;

  10:  using System.Threading.Tasks;

  11:  using Microsoft.Phone.Tasks;

12:

  13:  namespace SearchOn

  14:  {

  15:      public partial class MainPage : PhoneApplicationPage

  16:      {

  17:          public MainPage()

  18:          {

  19:              InitializeComponent();

20:

  21:              VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));

  22:          }

23:

  24:          protected override void OnNavigatedTo(NavigationEventArgs e)

  25:          {

  26:              base.OnNavigatedTo(e);

27:

  28:              if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))

  29:              {

  30:                  HandleVoiceCommand(NavigationContext.QueryString);

  31:              }

  32:              else if (e.NavigationMode == NavigationMode.New)

  33:              {

  34:                  NavigateToUrl(_defaultUrlForNewNavigations);

  35:              }

  36:              else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)

  37:              {

  38:                  NavigationService.GoBack();

  39:              }

  40:          }

41:

  42:          private void HandleVoiceCommand(IDictionary<string, string> queryString)

  43:          {

  44:              if (queryString["voiceCommandName"] == "searchSite" &&

  45:                  queryString.ContainsKey("siteToSearch"))

  46:              {

  47:                  SearchSiteVoiceCommand(queryString);

  48:              }

  49:          }

50:

  51:          private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

  52:          {

  53:              string text = await RecognizeTextFromWebSearchGrammar();

  54:              if (text != null)

  55:              {

  56:                  string siteName = queryString["siteToSearch"];

  57:                  await Speak(string.Format("Searching {0} for {1}", siteName, text));

58:

  59:                  string siteUrlTemplate = GetUrlTemplateFromSiteName(siteName);

  60:                  NavigateToUrl(string.Format(siteUrlTemplate, text, siteName));

  61:              }

  62:          }

63:

  64:          private async Task<string> RecognizeTextFromWebSearchGrammar()

  65:          {

  66:              string text = null;

  67:              try

  68:              {

  69:                  SpeechRecognizerUI sr = new SpeechRecognizerUI();

  70:                  sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);

  71:                  sr.Settings.ListenText = "Listening...";

  72:                  sr.Settings.ExampleText = "Ex. \"electronics\"";

  73:                  sr.Settings.ReadoutEnabled = false;

  74:                  sr.Settings.ShowConfirmation = false;

75:

  76:                  SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();

  77:                  if (result != null &&

  78:                      result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&

  79:                      result.RecognitionResult != null &&

  80:                      result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)

  81:                  {

  82:                      text = result.RecognitionResult.Text;

  83:                  }

  84:              }

  85:              catch

  86:              {

  87:              }

  88:              return text;

  89:          }

90:

  91:          private async Task Speak(string text)

  92:          {

  93:              SpeechSynthesizer tts = new SpeechSynthesizer();

  94:              await tts.SpeakTextAsync(text);

  95:          }

96:

  97:          private void NavigateToUrl(string url)

  98:          {

  99:              WebBrowserTask task = new WebBrowserTask();

 100:              task.Uri = new Uri(url, UriKind.Absolute);

 101:              task.Show();

 102:          }

 103:

 104:          private string GetUrlTemplateFromSiteName(string siteName)

 105:          {

 106:              return _siteUrlTemplateDictionary.ContainsKey(siteName)

 107:                  ? _siteUrlTemplateDictionary[siteName]

 108:                  : string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{1}");

 109:          }

 110:

 111:          private string _defaultUrlForNewNavigations = "http://m.bing.com/search?q=Rob's%20Rhapsody%20Windows%20Phone%208%20site:blogs.msdn.com";

 112:          private string _defaultUrlTemplateForUnknownSites = "http://m.bing.com/search?q={0}";

 113:

 114:          private Dictionary<string, string> _siteUrlTemplateDictionary = new Dictionary<string, string>()

 115:          {

 116:              { "Amazon", "http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" },

 117:              { "Bing", "http://www.bing.com/?q={0}" },

 118:              { "CNN", "http://www.cnn.com/search/?query={0}" },

 119:              { "Dictionary.com", "http://dictionary.reference.com/browse/{0}" },

 120:              { "Ebay", "http://www.ebay.com/sch/i.html?_nkw={0}" },

 121:              { "Facebook", "http://www.facebook.com/search/results.php?q={0}" },

 122:              { "Google", "https://www.google.com/search?hl=en&q={0}" },

 123:              { "Hulu", "http://www.hulu.com/#!search?q={0}" },

 124:              { "IMDB", "http://www.imdb.com/find?s=all&q={0}" },

 125:              { "Linked In", "http://www.linkedin.com/search/fpsearch?keywords={0}" },

 126:              { "MSN", "http://www.bing.com/search?scope=msn&q={0}" },

 127:              { "Twitter", "https://twitter.com/search/{0}" },

 128:              { "Weather", "http://www.weather.com/info/sitesearch?q={0}" },

 129:              { "YouTube", "https://www.youtube.com/results?search_query={0}" },

 130:              { "Zillow", "http://www.zillow.com/homes/{0}/" },

 131:          };

 132:      }

 133:  }

vcd.xml

   1:  xml version="1.0" encoding="utf-8"?>

2:

   3:  <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0">

4:

   5:    <CommandSet xml:lang="en-US">

6:

   7:      <CommandPrefix>Search OnCommandPrefix>

   8:      <Example>Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook, MSN, Twitter, Weather.com, YouTube, or ZillowExample>

9:

  10:      <Command Name="searchSite">

  11:        <Example>Amazon, Bing, CNN, Dictionary.com, Ebay, Facebook, MSN, Twitter, Weather.com, YouTube, or ZillowExample>

  12:        <ListenFor>{siteToSearch}ListenFor>

  13:        <ListenFor>{siteToSearch} dot comListenFor>

  14:        <Feedback>Search on {siteToSearch} for what?Feedback>

  15:        <Navigate Target="MainPage.xaml" />

  16:      Command>

17:

  18:      <PhraseList Label="siteToSearch">

  19:        <Item>AmazonItem>

  20:        <Item>BingItem>

  21:        <Item>CNNItem>

  22:        <Item>Dictionary.comItem>

  23:        <Item>EbayItem>

  24:        <Item>FacebookItem>

  25:        <Item>GoogleItem>

  26:        <Item>HuluItem>

  27:        <Item>IMDBItem>

  28:        <Item>KayakItem>

  29:        <Item>Linked InItem>

  30:        <Item>MSNItem>

  31:        <Item>NetflixItem>

  32:        <Item>TwitterItem>

  33:        <Item>Weather.comItem>

  34:        <Item>YouTubeItem>

  35:        <Item>ZillowItem>

  36:      PhraseList>

37:

  38:    CommandSet>

39:

  40:  VoiceCommands>

How many lines of code does it take to build a complete WP8 speech app?

Rob Chambers — Tue, 06 Nov 2012 15:31:18 GMT

In today's post, I'm going to show you how to use the three different parts of the speech platform on Windows Phone 8.0: Voice Commands, Speech Recognition, and Speech Synthesis. To do that, we're going to build an actual application, that has real value, but hopefully will be less than 100 lines of code.

Ready? Here we go ...

Here's the plan: Let's build an application that let's the user search Amazon.com for anything they'd like to find, all by simply using their voice. Here's the flow we'll enable today:

User: "Search On Amazon"
Phone: "What would you like to search for on Amazon?"
User: "Electronics" (or whatever they want)
Phone: "Searching Amazon for Electronics" (or whatever they said they wanted)

Sound interesting? Great.

First, let's start off by launching Visual Studio, and creating a new Windows Phone App project.

Then, we'll update the application's manifest, so we can use speech in our project.

On the Capabilities tab, click the checkmark next to the Microphone and Speech Recognition items. This tells the Market Place, at application submission time, that your app needs these capabilities. That way, the Market Place can be transparent to the end user, letting them know what your app is really going to do.

Next, let's add the XML file that specifies what voice commands you want users to be able to say. From the Solution Explorer, add a new item to your project, and select the Voice Command Definition item template.

The default template shows a few examples of what you can do. In a future post, I'll describe in more detail what each XML element does, and why they're important for your application.

For the app we're building today, simply type the following XML into the editor. As you're typing, notice that intellisense is active, making suggestions for you along the way, making it fairly easy to build your own XML file from the ground up.

   1:  xml version="1.0" encoding="utf-8"?>

2:

   3:  <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0">

4:

   5:    <CommandSet xml:lang="en-US">

6:

   7:      <CommandPrefix>Search OnCommandPrefix>

   8:      <Example>AmazonExample>

9:

  10:      <Command Name="searchSite">

  11:        <Example>AmazonExample>

  12:        <ListenFor>AmazonListenFor>

  13:        <Feedback>What would you like to search for on AmazonFeedback>

  14:        <Navigate Target="MainPage.xaml" />

  15:      Command>

16:

  17:    CommandSet>

18:

  19:  VoiceCommands>

Now that we have a Voice Command Definition file (often referred to as a VCD file), let's open up the MainPage.xaml.cs file to get ready to start writing some actual code! Now, go ahead and remove all the unnecessary comments. That'll help us achieve our 100 line count goal. :-)

Add the following using namespace statements; we need all of these for the code that follows. The first three are namespaces for the three areas of the speech platform on Windows Phone 8.0. The next namespace, System.Threading.Tasks, is needed so we can create our own asynchronous methods that we can call using the new C# await keyword. The last new namespace we'll use today, Microsoft.Phone.Tasks, will help us launch the WP8's version of Internet Explorer, allowing the user to go directly to Amazon's mobile website.

   1:  using Windows.Phone.Speech.VoiceCommands;

   2:  using Windows.Phone.Speech.Recognition;

   3:  using Windows.Phone.Speech.Synthesis;

   4:  using System.Threading.Tasks;

   5:  using Microsoft.Phone.Tasks;

Now, let's install the voice commands we've defined up above. To do that, add a line of code to the constructor of our page, after InitializeComponent, that will install our voice command definitions.

   1:  public MainPage()

   2:  {

   3:      InitializeComponent();

4:

   5:      VoiceCommandService.InstallCommandSetsFromFileAsync(

   6:          new Uri("ms-appx:///vcd.xml"));

   7:  }

Don't worry about keeping track of whether you've installed the VCD file before, or not, for now. The VoiceCommandService will do the right thing if you've already installed the voice commands. If you update the VCD file in your application, and hit F5, the new voice commands will be registered. But if nothing has changed since the last call to InstallCommandSetsFromFileAsync, the VoiceCommandService is smart enough to detect that no work is necessary.

Now that our voice commands are installed, how do we know when a user spoke one of our commands? Great question. That's where the existing OnNavigatedTo method comes in.

When a user speaks a voice command, the VoiceCommandService matches what the user said, with all the phrases you've previously registered when you called InstallCommandSetsFromFileAsync. If the VoiceCommandService determines that it's one of your commands, it will launch your application, on the page specified in your Navigate XML element's Target attribute.

So, let's go ahead and override the OnNavigatedTo protected method.

When OnNavigatedTo is called, we'll be able to tell it's a "New" navigation, as opposed to a "Back", or a "Refresh", by using the NavigationMode enumeration. The event args also contain the entire query string, broken up into an IDictionary of strings. If a query string parameter named "voiceCommandName" is present in that dictionary, you'll know that you've been launched by the VoiceCommandService. The value of that item in the dictionary will tell you which voice command the user has spoken.

So ... In our version of OnNavigatedTo, we'll check to see if it's a new navigation, and that if it's a voice command, and if so, we'll pass the query string dictionary off to another method to actually handle the voice command(s) there.

If OnNavigatedTo is being called for some other reason, you wouldn't normally do what I'm doing here. However, to keep our app as simple as possible, we're simply going to launch IE on Amazon.com's mobile web page.

Also, for this simple application, there's no reason for us to be on the back stack, so we'll try to get ourselves off the back stack. Since I don't know of a clean way to do that, I've opted to call GoBack on the NavigationService, which conceptually makes sense, but will actually throw an exception since we nothing is on the back stack. That's OK for this sample app, because it has the same desired effect.

   1:  protected override void OnNavigatedTo(NavigationEventArgs e)

   2:  {

   3:      base.OnNavigatedTo(e);

4:

   5:      if (e.NavigationMode == NavigationMode.New &&

   6:          NavigationContext.QueryString.ContainsKey("voiceCommandName"))

   7:      {

   8:          HandleVoiceCommand(NavigationContext.QueryString);

   9:      }

  10:      else if (e.NavigationMode == NavigationMode.New)

  11:      {

  12:          NavigateToUrl("http://m.amazon.com");

  13:      }

  14:      else if (e.NavigationMode == NavigationMode.Back &&

  15:                  !System.Diagnostics.Debugger.IsAttached)

  16:      {

  17:          NavigationService.GoBack();

  18:      }

  19:  }

Now, let's add the method to dispatch the voice commands. For now, we only have one voice command, but, like I said, we'll add more voice commands in the next post. Structuring the code this way up front, will save us a little time later.

   1:  private void HandleVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      if (queryString["voiceCommandName"] == "searchSite")

   4:      {

   5:          SearchSiteVoiceCommand(queryString);

   6:      }

   7:  }

OK. Now, let's actually do the site search. But, wait a minute We don't know what the user wants to search for yet. Right?

Let's pick up there, where the VoiceCommandService left off in the conversation with the user. After speaking, "Search On Amazon", they'll hear the prompt of "What would you like to search for on Amazon", due to our element in the VCD file for the searchSite .

So, let's add a new method to handle the conversation, starting at that point, breaking it up into 3 logical pieces.

First, we'll get some text from the speech recognizer. Then, if it recognized something, we'll let the user know what we're doing by telling them that we're about to do the search. Finally, we'll navigate to the site to actually do the search.

   1:  private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

   2:  {

   3:      string text = await RecognizeTextFromWebSearchGrammar();

   4:      if (text != null)

   5:      {

   6:          await Speak(string.Format("Searching Amazon for {0}", text));

   7:          NavigateToUrl(string.Format("http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}", text));

   8:      }

   9:  }

OK. Time to use the Speech Recognition APIs. Again, in a future post, I'll describe in more detail what all the various parameters and features, and why we have them as options in the namespace. But for now, just type the following code into Visual Studio:

   1:  private async Task<string> RecognizeTextFromWebSearchGrammar()

   2:  {

   3:      string text = null;

   4:      try

   5:      {

   6:          SpeechRecognizerUI sr = new SpeechRecognizerUI();

   7:          sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);

   8:          sr.Settings.ListenText = "Listening...";

   9:          sr.Settings.ExampleText = "Ex. \"electronics\"";

  10:          sr.Settings.ReadoutEnabled = false;

  11:          sr.Settings.ShowConfirmation = false;

12:

  13:          SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();

  14:          if (result != null &&

  15:              result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&

  16:              result.RecognitionResult != null &&

  17:              result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)

  18:          {

  19:              text = result.RecognitionResult.Text;

  20:          }

  21:      }

  22:      catch

  23:      {

  24:      }

  25:      return text;

  26:  }

Similarly, here's the method that we'll use to speak the feedback to the end user, using the built in SpeechSynthesizer. It's simpler than using the SpeechRecognizer. Synthesis, often called TTS, or Text-To-Speech, usually is more straight forward from a programmatic standpoint.

   1:  private async Task Speak(string text)

   2:  {

   3:      SpeechSynthesizer tts = new SpeechSynthesizer();

   4:      await tts.SpeakTextAsync(text);

   5:  }

And finally, here's the method we'll use to navigate to the specified URL.

   1:  private void NavigateToUrl(string url)

   2:  {

   3:      WebBrowserTask task = new WebBrowserTask();

   4:      task.Uri = new Uri(url, UriKind.Absolute);

   5:      task.Show();

   6:  }

That's it. 100 lines of code. Exactly! Not bad, eh? :-)

Oh yeah ... In my application, I've also removed nearly all the UI elements from the XAML file (MainPage.xaml), so it's not distracting to the user to see while the speech conversation is happening. Typically, you'd actually have a real page, but... This is just a 100 line sample. Right? :-)

OK. Time to try it out. If you haven't typed it all in yet, you can go ahead and ~~cheat~~ copy and paste the code from below. Be sure and remember to do the first few steps, declaring the capabilities required, otherwise you’ll get an Unauthorized exception when you try to call the Speech Recognition APIs.

Press F5, and you should land in the Emulator, on Amazon's web page.

Great. Our Voice Commands should be installed now.

Now, press and hold the Start button (the middle hardware button beneath the screen), and wait for the Speech "Listening..." screen to pop up. If you haven't run speech in the emulator yet, you'll need to accept the speech terms of use. Let's follow the script from above now:

You: "Search On Amazon"
Phone: "What would you like to search for on Amazon?"
You: "Electronics" (or whatever you want)
Phone: "Searching Amazon for Electronics" (or whatever you said you wanted)

Sometimes when you're using a fresh copy of the emulator, or if you haven't accepted the terms of use yet, you'll have to wait a few seconds for the voice commands to become active. If it doesn't work the first time, try try again. Or at least, try twice. :-)

That's it. You have now completed a reasonably useful speech enabled application for Windows Phone 8.0. In 100 lines of code.

Here's the full code listing:

using System;

using System.Collections.Generic;

using System.Net;

using System.Windows;

using System.Windows.Navigation;

using Microsoft.Phone.Controls;

using Windows.Phone.Speech.VoiceCommands;

using Windows.Phone.Speech.Recognition;

using Windows.Phone.Speech.Synthesis;

using System.Threading.Tasks;

using Microsoft.Phone.Tasks;

namespace SearchOn

    public partial class MainPage : PhoneApplicationPage

        public MainPage()

            InitializeComponent();

            VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));

        protected override void OnNavigatedTo(NavigationEventArgs e)

            base.OnNavigatedTo(e);

            if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))

                HandleVoiceCommand(NavigationContext.QueryString);

            else if (e.NavigationMode == NavigationMode.New)

                NavigateToUrl("http://m.amazon.com");

            else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)

                NavigationService.GoBack();

        private void HandleVoiceCommand(IDictionary<string, string> queryString)

            if (queryString["voiceCommandName"] == "searchSite")

                SearchSiteVoiceCommand(queryString);

        private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)

            string text = await RecognizeTextFromWebSearchGrammar();

            if (text != null)

                await Speak(string.Format("Searching Amazon for {0}", text));

                NavigateToUrl(string.Format("http://www.amazon.com/gp/aw/s/ref=is_box_?k={0}", text));

        private async Task<string> RecognizeTextFromWebSearchGrammar()

            string text = null;

try

                SpeechRecognizerUI sr = new SpeechRecognizerUI();

                sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);

                sr.Settings.ListenText = "Listening...";

                sr.Settings.ExampleText = "Ex. \"electronics\"";

                sr.Settings.ReadoutEnabled = false;

                sr.Settings.ShowConfirmation = false;

                SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();

                if (result != null &&

                    result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&

                    result.RecognitionResult != null &&

                    result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)

                    text = result.RecognitionResult.Text;

            catch

            return text;

        private async Task Speak(string text)

            SpeechSynthesizer tts = new SpeechSynthesizer();

            await tts.SpeakTextAsync(text);

        private void NavigateToUrl(string url)

            WebBrowserTask task = new WebBrowserTask();

            task.Uri = new Uri(url, UriKind.Absolute);

            task.Show();

Windows Phone 8 is here! Start speech enabling your apps…

Rob Chambers — Thu, 01 Nov 2012 14:30:00 GMT

It’s time to start speech enabling your Windows Phone applications, developers!

In today’s post, I’ll kick things off by consolidating and linking to a bunch of different resources that you can use to get started. I’ll also show you a real world application, that’s not a toy / sample app, that I’ve built using the same APIs you all have access to right now. From that, you can see what’s possible in a relatively short amount of time, and hopefully it’ll inspire you to think about what you can do in your applications. Over the coming weeks, I’ll dig into that application, piece by piece, and tell you exactly how you can do the same thing. We’ll probably end up with 10-15 posts total on the topic, having covered the basics in the first few posts, and covering some of the lower level details in the ones that follow.

So … Let’s start by taking a look at that real world application. It’s called Tivo Command. Take a look at the video, here, or embedded below, then continue on with the rest of post. :-)

Finished watching the demo of Tivo Command? Does it give you some ideas of things you could do in your apps? Want to add some speech features to your app?

OK. Here we go …

First, as a developer, you might want to read Kevin Gallo’s post from Tuesday this week, where he announced the Windows Phone 8 Developer Platform’s availability. Then, you probably should take a look at the Getting Started with Development for Windows Phone 8.0 guide on the new Windows Phone Dev Center. That’s where you’ll learn where you can download the SDK and tools, register your phone as a developer, and learn how to either build your first Windows Phone application, or migrate your existing Windows Phone 7 application to Windows Phone 8.

Next up, you probably want to take a look at What's new in the Windows Phone 8.0 SDK, to learn about all the cool new capabilities, like native code support, in app purchases, NFC support, custom app notifications and custom wallpaper. Of course, IMHO, the coolest feature of all of these is the new speech platform that all developers can take advantage of.

After reading the overview for Speech in Windows Phone 8, you’ll see that there are 3 main areas of investment that you can take advantage of in your apps running on Windows Phone 8: Voice Commands, Speech Recognition, and Speech Synthesis.

Last but not least, you should read thru the Speech Design Guidelines also on the Dev Center.

Of course, we also have all the APIs documented here:

That should be enough for one day, right? In upcoming posts I’ll start tearing apart Tivo Command v8 and show exactly how I added all the speech capabilities, using the Windows Phone 8.0 SDK.

Let me know if you have any questions in the comments below…

Developers: Coming soon to a phone near you… Windows Phone 8

Rob Chambers — Wed, 24 Oct 2012 12:54:25 GMT

Windows Phone 8 is almost here!!

Joe Belfiore introduced Windows Phone 8 to the world in June, 2012.
Kevin Gallo summarized what’s new for developers, just a week later.
The new Windows Phone Dev Center came online in August.
The Windows Phone 8 SDK Preview program was announced in September.
The Windows Phone 8 press event is set for Monday, October 29th, 2012.

Are you ready to get started?

Fellow developers…

Do you want your users to be able to quickly, naturally, and easily access your application’s most used features?
Do you want your application to be seamlessly integrated into the phone’s existing user experience?
Wouldn’t it be awesome to do that with the same Visual Studio tools you use today to make apps?
Don’t you wish you could do all that, in just a few minutes, instead of weeks, months, or years?

If you answered yes to all those questions, it sounds like you want to use our latest and greatest speech platform, included in Windows Phone 8. It’s the same platform we’ve used to enable Windows Phone 8’s built in speech experiences, like Voice Dialing, SMS play back and reply, and even Bing’s very own Voice Search.

Are you ready to get started? Yes? I’m ready to start showing you how …

Stay tuned; check back next week. I think you’ll be glad you did.

:-)

Windows Phone 7.5 (Mango) and Speech Recognition

Rob Chambers — Tue, 13 Sep 2011 09:49:41 GMT

Here at Microsoft we've been busy over the last year adding cool new speech features to a bunch of Microsoft platforms and applications. You've undoubtedly heard about at least one of these recently (unless of course you live under a rock :-])...

One such example is the work we've done in Windows Phone 7.5 (aka Mango). Perhaps the sweetest new speech feature we've added to Windows Phone is the work we've done around the end to end experience of text messages. Instead of telling you about it myself, you should take a look at a recent article on ZDNet written by Matthew Miller. Check it out here.

As always, we'd love to hear what you think, so drop us an email... We're listening...

Want to see what you can do with Speech across 3 screens and a cloud?

Rob Chambers — Thu, 14 Jan 2010 12:24:00 GMT

Happy 2010!

I talked about what we’ve all been up to in 2009 in the Speech group on the Speech Team blog the other day… Focusing across the 3 screens and a cloud...

I also thought it’d be interesting to see the demo/video I put together on Channel 10 that demos some of that, and talks about how developers can take advantage of our speech platform. You can check it out here.

Let us know if you have any questions.

Bing on the iPhone (with Voice Search!)

Rob Chambers — Wed, 16 Dec 2009 08:45:59 GMT

Microsoft's Bing for iPhone went live yesterday, and is now available in the iPhone application store. You can read more about it here on PC World.com. I downloaded the app last night on my wife's iPhone and it's working great!

The Speech team here at Microsoft has been working with Bing on the Voice Search capabilities for a while now, and the whole team is excited that the latest version of our speech technology is powering the newest version of Bing's mobile client software.

I especially like how easily you can get to movies, stocks, and images, with simple voice commands. Download the app, and try it out for yourself.

Let us know what you think of the Voice Search capabilities!