WSR Macro of the Day 2008.05.23: "Symbol That"
Now that the WSR Macro tool has been out for a couple weeks, it's clear that people are loving it! For those of you that haven't tried it, I'm going to start posting example macros to help everyone realize just how powerful the system really is.
Today's macro is called "Symbol that".
A long time speech user and Microsoft Speech enthusiast, Itamar Even-Zohar, asked me quite a long time ago to help him with a feature he wanted in Microsoft Word. He wanted to be able to put quotes around whatever is selected. Prior to the release of WSR Macros, I made a VBA macro inside word for him that he hooked up to a button on Word 2003's toolbar. Because he uses SR to control Word and everything else in the OS, he could name the button "Quote that", and he'd be able to select something, and then say "Quote that".
Now that the Macro tool is out, he asked me what that'd look like if I ported it to the new WSR Macro tool. Let's start with the most basic form, that does exactly what Itamar wanted:
<speechMacros>
<command>
<listenFor>quote that</listenFor>
<sendKeys>{250 WAIT}{{ctrl}}c{u+0027}{{ctrl}}v{u+0027}</sendKeys>
</command></speechMacros>
Now, with this macro, whenever I say "Quote that", the system will wait 250ms, press control-c (aka Copy), then press the quote key (that's what {u+0027} is in hexadecimal Unicode format), then press control-v (aka Paste), and then another quote. It not only works in Word, but all applications that have cut and paste edit controls (which is pretty much everything).
Pretty easy, eh?
Well, what if I also wanted to do that with double quotes? Here's what that'd look like:
<speechMacros>
<command>
<listenFor>quote that</listenFor>
<sendKeys>{250 WAIT}{{ctrl}}c{u+0027}{{ctrl}}v{u+0027}</sendKeys>
</command><command>
<listenFor>double quote that</listenFor>
<sendKeys>{250 WAIT}{{ctrl}}c{u+0022}{{ctrl}}v{u+0022}</sendKeys>
</command></speechMacros>
You can see there are two commands now; the first is the same as it was, and the second one is pretty similar, it just changes what is being listened for, and also changes what the surround characters are. Now, I could keep adding these one right after another ... or I could do something much cooler... Let's do that!
How about just having one command, but letting it be parameterized with the type of symbols to be used? Here's what the would look like:
<speechMacros>
<command>
<listenFor> [symbol] that</listenFor>
<sendKeys>{250 WAIT}{{ctrl}}c {[symbol.keys]} </sendKeys>
</command><listenForList name="symbol" propname="keys">
<item propval="{u+0027}{{ctrl}}v{u+0027}">quote</item>
<item propval="{u+0027}{{ctrl}}v{u+0027}">quotes</item>
<item propval="{u+0022}{{ctrl}}v{u+0022}">double quote</item>
<item propval="{u+0022}{{ctrl}}v{u+0022}">double quotes</item>
<item propval="{u+005b}{{ctrl}}v{u+005d}">?square bracket</item>
<item propval="{u+005b}{{ctrl}}v{u+005b}">?square brackets</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">squiggly bracket</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">squiggly brackets</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">curly bracket</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">curly brackets</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">paren</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">parens</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">parenthesize</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">parentheses</item>
<item propval="{u+003c}{{ctrl}}v{u+003e}">less ?than greater ?than</item>
<item propval="{u+003c}{{ctrl}}v{u+003e}">less ?than greater thans</item>
</listenForList></speechMacros>
Now, this is a little more complex, but it's actually smaller than it would be if I had a command for each way to say this. I've done several things here that I'll describe in some more detail.
First, I went back to a single command, but I've changed what you need to say. Instead of embedding the name of the symbol in the command, I'm referring to a list of things I could say. In this example, I'm referring to the list (or rule in speech nomenclature) named "symbol".
Second, I defined the list of phrases for the "symbol" rule using the <listenForList> element. This allows me to say things like "quote that", "double quote that", "curly bracket that", etc...
Lastly, I've also given that list rule a property name that I can use to look up properties for each of the phrases. I'm doing this so I can have one generic action in the command, and have that action parameterized for each symbol. That's what the {[symbol.keys]} does. It looks up the semantic property named "keys" defined in the rule named symbol, for what I actually have spoken to invoke the rule.
You can see that each of those semantic properties also includes a {{ctrl}}v (Paste) in between the symbols. I did that so I could also have something like "Add a comma after that", and it's semantic property would be "{{ctrl}}v, ". Get it? That way the symbol doesn't have to appear both before and after...
Using semantic properties is pretty powerful. It also allows me to have separate ways of invoking the rule, and I don't have to worry about where in the phrase the semantic property appears. So, for example, I can add another way to invoke the rule by changing the command to look like this:
<command>
<listenFor>[symbol] that</listenFor>
<listenFor>add [symbol] to that</listenFor>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command>
Now, I can say "Add double quotes to that", or "Double quote that", and the right thing will happen. This is useful for additional phrases, and it's also very useful when you're supporting multiple languages (something the WSR Macro tool doesn't do quite yet).
Great, now I've not only done what Itamar needed, but I've extended it to be generic for all type of "Symbol that" operations.
Hmmm... But that's not good enough for me. The macro system is super powerful, so let's add some more to it.
In the built-in experience in Vista, you can also say things like "Select the last 2 words", or "Select the next three characters". What if I wanted to make this macro do similar things, but for adding symbols? Like, "Double quote the next three words". Sound fun? Let's try ...
Let's add another command:
<command>
<listenFor>[symbol] [something] </listenFor>
<emulateRecognition>select {[something]}</emulateRecognition>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command>
And, let's add another rule or two that let's me specify that "something":
<rule name="something">
<list>
<p>
<o>the</o>
<o>
<list>
<p>previous</p>
<p>next</p>
</list>
<o><ruleref name="1to20"/></o>
</o>
<list>
<p>character</p>
<p>characters</p>
<p>word</p>
<p>words</p>
<p>sentence</p>
<p>sentences</p>
<p>paragraph</p>
<p>paragraphs</p>
<p>document</p>
</list>
</p>
</list>
</rule><numbers name="1to20" start="1" stop="20"/>
The syntax for the "something" rule is actually just the normal XML syntax that the speech platform already uses for defining rules for the recognizer. You can read about it here on MSDN.
The syntax for the "1to20" numbers rule should be pretty self explanatory. The combination of the two of these allows me to say pretty much anything I could say in Vista to select text, but it allows me to do it with my own phrases for "[Symbol] [something]".
So, now when I say "Double quote the next three words", the command we just added will first emulate me saying "Select the next three words", and then it'll go ahead and do the same thing the previous command did with the ctrl-c and send keys defined for the symbol operation.
OK... That's pretty cool, but what about being able to refer to the text in the document itself. In Vista, I can say things like "Select that's pretty cool", and it'll select those words in this paragraph. OK... We can make the macro do that kind of operation as well. Check it out:
<command>
<listenFor>[symbol] [textInDocument] </listenFor>
<listenFor>[symbol] the word [textInDocument]</listenFor>
<emulateRecognition waitForDisambiguation="15">select {[textInDocument]}</emulateRecognition>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command>
Now, if I say "Double quote that's pretty cool", we'll emulate "Select that's pretty cool", and then do the same thing as before.
Note the "waitForDisambiguation" attribute. That tell's the macro system to wait and see if what we just emulated would need to be disambiguated by the system. If it is, let the user complete the disambiguation (only giving the user 15 seconds to complete it), and then carry on with the rest of the macro. If the user cancels out of the disambiguation, the command will be terminated early.
Wow. This is all very cool. At least I think so ... :-) But Wait! There's more!
What if I say "Double quote sentence", and the word "sentence" is actually in the document. That makes the system ambiguous... Which one do I mean? Do I want the word "sentence" double quoted, or do I actually want the current sentence double quoted? To help the system be deterministic, we can introduce the use of priorities. Check it out:
<command priority="3" >
<listenFor>[symbol] that</listenFor>
<listenFor>add [symbol] to that</listenFor>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command><command priority="2" >
<listenFor>[symbol] [something]</listenFor>
<emulateRecognition>select {[something]}</emulateRecognition>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command><command priority="1" >
<listenFor>[symbol] [textInDocument]</listenFor>
<listenFor>[symbol] the word [textInDocument]</listenFor>
<emulateRecognition waitForDisambiguation="15">select {[textInDocument]}</emulateRecognition>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command>
Now, deterministically, the system will favor working with the current selection first (the that commands), then the unit based selection model (the [something] command), then the text based model (the [textInDocument] command).
OK... OK... Can we make this any more cool? Yep.
What if I say "Double quote the dog has fleas", but my document doesn't actually contain that text? Well ... The system will probably think that's dictation, and it'll inject that text into the document. That's likely not what you want. So, let's listen for the user saying "[Symbol] *", which is to say, let's listen for "[Symbol]" followed by anything the user might say (that's what the "*" syntax means). That looks like this:
<command priority="0">
<listenFor>[symbol] *</listenFor>
<setTextFeedback style="warning">What was that?</setTextFeedback>
</command>
Now, if I say something that's not matched by the other 3 commands, it'll get caught by this command, and it'll set the text feedback in the Vista microphone UI to "What was that?" in the warning style. This is what the normal experience does when you say something that it doesn't understand at all. Neat...
Do you want to add one more thing? Well, I do...
Let's extend this to do formatting as well, for applications like Wordpad and Microsoft Word. You know, like bold, italics, underline? How could we do that? Pretty easy, really. Add 4 more commands that look like this:
<command priority="3">
<condition operator="or">
<appIsInForeground processName="winword.exe"/>
<appIsInForeground processName="wordpad.exe"/>
</condition>
<listenFor>?format [format] that</listenFor>
<sendKeys>{250 WAIT}{[format.keys]}</sendKeys>
</command><command priority="2">
<condition operator="or">
<appIsInForeground processName="winword.exe"/>
<appIsInForeground processName="wordpad.exe"/>
</condition>
<listenFor>?format [format] [something]</listenFor>
<emulateRecognition>select {[something]}</emulateRecognition>
<sendKeys>{250 WAIT}{[format.keys]}</sendKeys>
</command><command priority="1">
<condition operator="or">
<appIsInForeground processName="winword.exe"/>
<appIsInForeground processName="wordpad.exe"/>
</condition>
<listenFor>?format [format] [textInDocument]</listenFor>
<listenFor>?format [format] the word [textInDocument]</listenFor>
<emulateRecognition waitForDisambiguation="15">select {[textInDocument]}</emulateRecognition>
<sendKeys>{250 WAIT}{[format.keys]}</sendKeys>
</command><command priority="0">
<listenFor>[format] *</listenFor>
<setTextFeedback style="warning">What was that?</setTextFeedback>
</command>
Notice that the first three of these specify that the command should only work in the winword.exe and wordpad.exe processes. That's because notepad doesn't know how to deal with formatting. But ... The last command listens for "[format] *" in all processes, so even if I forget and say "Bold that" when I'm in notepad, this last command will catch it and give me feedback.
And, you'll also need a "format" list rule now. It looks like this:
<listenForList name="format" propname="keys">
<item propval="{{ctrl}}b">bold</item>
<item propval="{{ctrl}}i">italicize</item>
<item propval="{{ctrl}}u">underline</item>
</listenForList>
So, now if we put the whole macro together, it looks like this:
<speechMacros>
<command priority="3">
<listenFor>[symbol] that</listenFor>
<listenFor>add [symbol] to that</listenFor>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command><command priority="2">
<listenFor>[symbol] [something]</listenFor>
<listenFor>add [symbol] to [something]</listenFor>
<emulateRecognition>select {[something]}</emulateRecognition>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command><command priority="1">
<listenFor>[symbol] [textInDocument]</listenFor>
<listenFor>[symbol] the word [textInDocument]</listenFor>
<listenFor>add [symbol] to [textInDocument]</listenFor>
<listenFor>add [symbol] to the word [textInDocument]</listenFor>
<emulateRecognition waitForDisambiguation="15">select {[textInDocument]}</emulateRecognition>
<sendKeys>{250 WAIT}{{ctrl}}c{[symbol.keys]}</sendKeys>
</command><command priority="0">
<listenFor>[symbol] *</listenFor>
<setTextFeedback style="warning">What was that?</setTextFeedback>
</command><command priority="3">
<listenFor>?format [format] that</listenFor>
<sendKeys>{250 WAIT}{[format.keys]}</sendKeys>
</command><command priority="2">
<listenFor>?format [format] [something]</listenFor>
<emulateRecognition>select {[something]}</emulateRecognition>
<sendKeys>{250 WAIT}{[format.keys]}</sendKeys>
</command><command priority="1">
<listenFor>?format [format] [textInDocument]</listenFor>
<listenFor>?format [format] the word [textInDocument]</listenFor>
<emulateRecognition waitForDisambiguation="15">select {[textInDocument]}</emulateRecognition>
<sendKeys>{250 WAIT}{[format.keys]}</sendKeys>
</command><command priority="0">
<listenFor>[format] *</listenFor>
<setTextFeedback style="warning">What was that?</setTextFeedback>
</command><listenForList name="symbol" propname="keys">
<item propval="{u+0027}{{ctrl}}v{u+0027}">quote</item>
<item propval="{u+0027}{{ctrl}}v{u+0027}">quotes</item>
<item propval="{u+0022}{{ctrl}}v{u+0022}">double quote</item>
<item propval="{u+0022}{{ctrl}}v{u+0022}">double quotes</item>
<item propval="{u+005b}{{ctrl}}v{u+005d}">?square bracket</item>
<item propval="{u+005b}{{ctrl}}v{u+005b}">?square brackets</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">squiggly bracket</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">squiggly brackets</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">curly bracket</item>
<item propval="{u+007b}{{ctrl}}v{u+007d}">curly brackets</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">paren</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">parens</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">parenthesize</item>
<item propval="{u+0028}{{ctrl}}v{u+0029}">parentheses</item>
<item propval="{u+003c}{{ctrl}}v{u+003e}">less ?than greater ?than</item>
<item propval="{u+003c}{{ctrl}}v{u+003e}">less ?than greater thans</item>
</listenForList><listenForList name="format" propname="keys">
<item propval="{{ctrl}}b">bold</item>
<item propval="{{ctrl}}i">italicize</item>
<item propval="{{ctrl}}u">underline</item>
</listenForList><rule name="something">
<list>
<p>
<o>the</o>
<o>
<list>
<p>previous</p>
<p>next</p>
</list>
<o><ruleref name="1to20"/></o>
</o>
<list>
<p>character</p>
<p>characters</p>
<p>word</p>
<p>words</p>
<p>sentence</p>
<p>sentences</p>
<p>paragraph</p>
<p>paragraphs</p>
<p>document</p>
</list>
</p>
</list>
</rule><numbers name="1to20" start="1" stop="20"/>
</speechMacros>
I hope you found this post useful. If you haven't downloaded the new Macro tool, try it out. Let us know what you think if the tool, this macro, or what you'd like to see as part of future "Macro of the day" posts.