In addition, we’re very excited to see the release of Python Tools for Visual Studio (PTVS) 1.0 by our friends on the Python Tools team, which works as a plug-in for Visual Studio 2010; even if you don’t have Visual Studio you can get the free VS shell and use PTVS with that (details are on the PTVS install page).  For years, users have been asking us for a great IDE for Sho, and we’re glad to finally have an answer for them. PTVS makes for a fantastic IDE for Python in general (standard Python as well as IronPython), including powerful debugging capabilities, an editor with syntax highlighting, intellisense, find all references, find definition, and other such great utilities, as well as a very nice interactive window (REPL).  In addition, we’ve done some work to make Sho work smoothly with PTVS, and our new IDE page explains how to install and set it all up. 

You can get the latest bits and find links to the Sho pages mentioned above at http://research.microsoft.com/sho

Sho 2.0.5 changelist:

• IronPython 2.7 and 2.6 versions
• Smooth animations in Sho plots via Double Buffering (via the ShoChart.Freeze command)
• Upgraded MKL to version 10.3.4
• New ArrayMathInPlace class: implements the same elementwise operations as ArrayMath, but does them in-place on the input array
• ShoPlotHelper.Grid() now returns a ShoChartGrid object instead of just a Control
• Fixed Sho.py so scripts run from ipy.exe that import Sho work properly (they used to run twice)
• Fixed a bug in the slicing of multidimensional arrays
• Fixed a bug in Multiply Into
• Fixed console crash when tab-completing certain types
• Fixed doc() crash when called on certain types

As you may know, the Kinect is a pretty complex sensor, and the SDK adds even more layers of sophistication. The hardware itself is comprised of a depth sensor and a microphone array; the SDK extracts skeletons of multiple people in the scene, high-quality audio output, and sound source localization (estimates of where the sound is coming from). In this post, we’re only going to work with the skeleton data; if there’s demand (i.e., if we hear from you!), we can add more posts to get the other sorts of data as well.

What Can It Do?

It’s often best to start with a demo. Let’s start with something basic: let’s use Sho’s built-in plotting functionality to just plot all the available skeletons and label all the individual joints. Kinect.py has a KinectTracker class which uses a polling model to get skeleton information. The class’ skeletonFrame member will contain the most recent set of skeletons (or None if no skeleton has been found yet), and skeletonUpdateTime will be set to the last time the skeletons were updated. To make it a bit more fun, we’ll have the skeletons update in real time within the plot (see the video). The video shows Sho’s new double buffering feature for plots, which will appear in Sho 2.0.5 (to be released in the next few weeks); in the current version of Sho you’ll see some “flashing” when updating the plot.

The next question, of course, is how we did that and how much code it took. Fortunately it’s quite simple and not very much code at all:

That example demonstrates all the important stuff: how to get at individual joints and their 3D locations, as well as how the joints are connected together. From there, it’s easy to add a few more lines of code and create simple virtual 3D objects. In the example below, I’ve created a simple staff object with a fixed 3D length that’s centered between the two hands. The code below can be inserted right before the “draw joints” section in the bodyplot() function above:

When the hands get within a certain distance of each other, the staff (magically) appears, and our little stick figure friend can do some fancy tricks. A screenshot is below, but you’ll want to click here to see the video.

How It Works

The glue code that gets the SDK working in Sho, kinect.py, is also quite short, but what it does is a little complicated. Since the managed layer of the Kinect SDK depends on the WPF threading model and message pump, we have to create a shim WPF application and window that will get the events. That application has a callback that’s called every time it gets new skeleton data; inside that callback we save the skeletonFrame information and the skeletonUpdateTime in the KinectTracker object, which we can access outside of the shim application’s thread. The entirety of kinect.py is below; note you may need to update the paths to point to wherever you installed the SDK:

How To Get Started with Your Own Code

To develop your own Kinect demos, you’ll have to start by getting yourself a Kinect sensor. I’d recommend the standalone sensor, since it comes with a power supply; if you get the package with an xbox 360 you’ll need to buy a separate power supply as it only comes with a cable to connect to the 360 console. Once you have the device, you’ll need to install the Kinect SDK. Once you’ve done that, reboot your machine, plug in your kinect, and you should be ready to go.

In order to use the sensor from Sho, you’ll have to use the 32-bit console, shoconsole32, even on a 64-bit machine, as although the drivers work on both 32-bit and 64-bit machines, the managed SDK only supports calls from 32-bit contexts. You’ll also want to save the glue code above as kinect.py. Once in the console, import it and create a KinectTracker object with startKinectTracker:

>>> import kinect
>>> kt = kinect.startKinectTracker()

Once a skeleton is found in the sensor’s range, its data will be copied to kt.skeletonFrame and you can use it as in the demos above.

And You’re Off!

So there you have it; you now know how to talk to the Kinect SDK from Sho using the wrapper in kinect.py above. I’m sure you’ll do far more creative things than adding a red staff to a stick figure, and we’d love to hear about it. Please send us links to your own demo videos in the comments!

Start by downloading the data – pick the “Historical” option and click “Get Full Set;” this will give you a zip file which will contain a single file, sp500hst.txt.  Save that to some handy directory.  If we open up that file in a text editor, we’ll see a bunch of lines like the below:

20100310,CBS,14.45,14.75,14.39,14.75,81612
20100311,CBS,14.63,14.8,14.4,14.8,65851
20100312,CBS,14.79,14.85,14.42,14.6,66466
20100315,CBS,14.52,14.535,14.22,14.44,68313
20100316,CBS,14.44,14.78,14.31,14.78,103030
20100317,CBS,14.83,14.945,14.46,14.5,80747
20100318,CBS,14.51,14.655,14.24,14.37,55267
20100319,CBS,14.42,14.63,13.98,13.98,99261
20100322,CBS,13.96,14.14,13.7,14.07,58456
20100323,CBS,14.14,14.28,14.06,14.26,66844
20100324,CBS,14.13,14.43,14.03,14.24,84188
20100325,CBS,14.42,14.44,13.85,13.87,108878

In other words, the date, followed by the symbol, followed by open, high, low, close, and the volume for that day.  We’re interested in grabbing the closing value for each day for a given stock so that we can plot it, compute running averages, make predictions, etc.  Let’s write a function that will extract this from the file:

def getseries(symbol, filename):
    data = csvreadArray(filename)
    dates = []
    prices = []
    for line in data.Rows:
        if line[1] == symbol:
            dates.Add(line[0]) # date
            prices.Add(float(line[5])) # closing price
    return(dates,DoubleArray.From(prices))   

This will get us the dates and closing price for a given symbol.  Note that we convert the list of prices to a DoubleArray for ease of later manipulations. Let’s retrieve the prices for symbol “CBS” and plot the result:

>>> dates, prices = getseries("CBS", "c:/mypath/sp500hst.txt")
>>> plot(prices)

That gives us a first look at the raw data.  Note that the x axis labels are just the index number of each element; we’d rather see the appropriate date labels for the x axis, but since they’re strings and not numeric values, we can’t just pass them in via plot(x,y).  We can instead modify the x-axis labels with the xlabels command – we just need to tell it which locations to label and what the respective values would be, so we’ll create a series of numbers spaced by 10 and then pick out the appropriate labels with a list comprehension:

>>> sd = range(0,len(dates),10)
>>> xlabels(sd, [dates[ind] for ind in sd], orientation=90) # create date labels on x axis and turn them 90 degrees

Now we’d like to see the moving averages over different windows for the stock in addition to its daily value.  We can use the signal processing tools in Sho to easily compute a moving average by convolving the time series with a box filter, which we can represent via ones(N,1)/N where N is the length of the filter.  We divide by N to normalize the sum of the filter to 1.  We’ll compute a 10-day and a 30-day moving average.

>>> smooth10 = SignalProc.conv(DoubleArray.From(prices).T, ones(10,1)/10.0, CompMode.Auto, BoundaryMode.Reflect)
>>> smooth30 = SignalProc.conv(DoubleArray.From(prices).T, ones(30,1)/30.0, CompMode.Auto, BoundaryMode.Reflect)

We’ll now plot these smoothed signals on top of the original prices by using the hold() command to hold the previous plot.  Finally, we’ll add a legend to label the separate time series; we’ll also add a title for good measure.

>>> hold()
>>> plot(smooth10)
>>> plot(smooth30)
>>> legend('daily prices','10-day average','30-day average')
>>> plottitle('Daily Prices for CBS')

That gets us to the plot below:

In the next post, we’ll show how build a simple autoregressive model for predicting future prices (though not with particularly high accuracy).

>>> clr.AddReference("System.Speech")
>>> from System.Speech.Synthesis import SpeechSynthesizer
>>> synth = SpeechSynthesizer()
>>> synth.Speak("Hello all and welcome to the Blog of Sho")

That’s all it takes to get Sho talking back to you.  If nothing else, this is a fun way to get Sho to tell you when it’s finished with a long optimization run :).

To get started, you’ll need to have some kind of microphone.  Headsets work best, but webcam mics, built-in laptop mics, or analog mics will work as well.  Once you’ve got that plugged in, you need to do a little bit of setup:

>>> clr.AddReference("System.Speech")
>>> from System.Speech.Recognition import *

Now we’ll instantiate a speech recognition engine.  We’ll set it up with the default dictation grammar – you can custom design a grammar as well, but the default works pretty well for general use.

>>> sre = SpeechRecognitionEngine()
>>> sre.SetInputToDefaultAudioDevice()
>>> sre.LoadGrammar(DictationGrammar())

Now we can tell the recognizer to listen for speech.  Type in the following line and say something into the mic, like “a family of ducks can create a traffic jam:”

>>> res = sre.Recognize()

This call will block until it gets an utterance from the microphone.  Once it returns, you can print the resulting text:

>>> print res.Text

You can also use the recognizer in an asynchronous mode, where each speech event will trigger a callback.  To do this, first define a function and hook it to the recognizer’s SpeechRecognized event:

>>> def printspeech(sender, event):
                txt = e.Result.Text
                print txt
>>> sre.SpeechRecognized += printspeech

Then start up the recognizer in asynchronous mode; the call below will return immediately:

>>> sre.RecognizeAsync(RecognizeMode.Multiple)

The printspeech() function will be called after each speech utterance. If you leave out the RecognizeMode.Multiple, the recognizer will stop listening after the first event.  Note that the callback can be a method in an object, which will allow you to keep some state.  In this way, you can create all kinds of speech-enabled applications: a voice-controlled chess game, a drawing program that lets you change pens by voice, etc.  Give it a shot and let us know what you build with it!

>>> ShoLoadAssembly("c:/Program Files/Microsoft Biology Initiative/1.0/MBF/MBF.dll")
>>> import MBF
>>> filename = "c:\\path\\to\\your\\file.fa"
>>> parser = MBF.IO.SequenceParsers.FindParserByFile(filename)
>>> sequences = parser.Parse(filename)
>>> for x in sequences: print x

CCAGGCTTTAGGTATCACCACTAAAATCTCCAGGGGCAC
TTTCCAGGCTTTAGGTATCACCACTAAAATCTCAAG
GGTATCACCACTAAAATCTCCAGGGGCACCATTGAA
...

 There's of course much more you can do with the MBF tools than just read sequences from files - you can parse sequences, make visualizations, do sequence alignment, multi-sequence alignment, fragment assembly, transcription, translation, pattern matching against sequences, and more (see their MBF training page for more information), all from the .NET language of your choice.  All this functionality is thus available in Sho; using the code is as simple as the small example above.

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using ShoNS.Visualization;
using ShoNS.Array;
using ShoNS.MathFunc;

namespace testForm
{
    // Create a composite visualization by deriving from System.Windows.Forms.Form
    public class testForm : Form
    {
        private ShoChart ch;     // this will hold a Sho contour chart
        private Button button1;  // a button to update the contour chart
        private Label label1;    // a label to show how many times the button has been clicked
        private int num;         // a counter


        public testForm()
        {
            num = 0;
            InitializeComponent();
        }

        // code to generate (slightly random) data and create a contour chart from it
        // this method shows examples of calling into the Sho math libraries and the Sho viz libraries
        private ShoChart GenChart()
        {
            // first, create a range object that spans from 0 to pi, with 100 elements
            DoubleRange b = new DoubleRange(0.0, Math.PI);
            b.Count = 100;
            // now, create a row vector and fill it with random numbers
            DoubleArray x = new DoubleArray(1, 100);
            ArrayRandom.FillRandom<double>(x);
            // shift and scale the random numbers, and add sin(b) to it, to make a noisy sinusoid
            x = (x-0.5) * 0.1 + ArrayMath.Sin(b);
            // analogously, we'll make a noisy cosine row vector, 100 long
            DoubleArray y = new DoubleArray(1, 100);
            ArrayRandom.FillRandom<double>(y);
            y = (y-0.5) * 0.1 + ArrayMath.Cos(b);
            // now, make a matrix from the outer product of x and y.
            DoubleArray arr = x.T * y;
            // from the matrix, create a contour plot. Second argument says automatically choose the contours
            ShoChart ch = new ShoContourChart(arr, null);
            // all of the Sho viz classes inherit from control, so we can set WinForms control parameters
            // to size the contour plot and place it within the parent form.
            ch.AutoSize = true;
            ch.Location = new Point(25, 75);
            ch.Name = "ContourChart";
            ch.Size = new Size(375, 225);
            return ch;
        }

        // a button click will create a new contour chart with slightly randomized data
        private void buttonClick(object sender, EventArgs e)
        {
            SuspendLayout();
            // take the old contour plot out of the form
            Controls.Remove(ch);
            // generate a new one, and put it into the form
            ch = GenChart();
            Controls.Add(ch);
            // increment the label
            label1.Text = num.ToString();
            num++;
            // relayout and refresh the form
            ResumeLayout(false);
            PerformLayout();
            Refresh();
        }

        private void InitializeComponent()
        {
            // code derived from form wizard: create a button and label
            button1 = new Button();
            label1 = new Label();

            SuspendLayout();

            // position and parameterize the button
            button1.Location = new Point(175, 50);
            button1.Name = "button1";
            button1.Size = new Size(100, 20);
            button1.TabIndex = 0;
            button1.Text = "click me";
            button1.UseVisualStyleBackColor = true;
            button1.Click += new EventHandler(buttonClick); // hook up button click handler

            // position and parameterize the label
            label1.AutoSize = true;
            label1.Location = new Point(215, 25);
            label1.Name = "label1";
            label1.Size = new Size(50, 20);
            label1.Text = num.ToString();
            label1.TextAlign = ContentAlignment.MiddleCenter;
            num++;

            // create a new contour plot
            ch = GenChart();

            AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
            AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
            ClientSize = new System.Drawing.Size(425, 325);
            Controls.Add(label1);
            Controls.Add(button1);
            Controls.Add(ch);
            Name = "Form1";
            Text = "Example showing ShoChart embedded in WinForm using C#";

            // layout the controls: now we're ready to start the message pump
            ResumeLayout(false);
            PerformLayout();
        }
    }
}

From inside of Sho, you can test this by dynamically loading the dll, and running the form in its own thread (made easy with ShoThread):


>>> load('testForm.dll')
<Assembly testForm, Version=1.0.0.0, Culture=neutral, PublicKeyToken=...>
>>> from testForm import *
>>> t = ShoThread(lambda : System.Windows.Forms.Application.Run(testForm()))
>>> t.Start()

To debug your code, you can still use the Sho Console - just attach the debugger to the shoconsole.exe process, set your breakpoints, and continue as normal.  When your breakpoints are hit, execution will stop and you can step through code, inspect variables, and all the rest.  You can see a walkthrough of the debugging process in this video I put up on the Sho Videos page.