Web Applications

  • Article

Kinect for Windows 1.8

The Kinect for Windows SDK JavaScript APIs give HTML5 applications access to Kinect data for interactions and visualization. This allows HTML5 applications running in a browser to connect to the sensor through a server running on the local computer. You can use this to create kiosk applications on dedicated computers. The web server component is a template that can be used as-is or modified as needed. For more information about the JavaScript APIs, see JavaScript Reference.

Note

The Kinect for Windows SDK JavaScript APIs support Internet Explorer 10 and later, Mozilla Firefox, and Google Chrome.

The following sections demonstrate how to use the Kinect for Windows SDK JavaScript APIs.

  1. Include the JavaScript source files
  2. Initialize the Kinect sensor (optional)
  3. Retrieve an instance of the KinectSensor interface
  4. Enable and configure the Kinect sensor streams
  5. Add a handler to process incoming stream frames
  6. Hook up the user interaction stream
  7. Display real-time bitmap streams
  8. Add a handler to respond to events from the sensor (optional)
  9. Extend functionality with custom streams (optional)

1. Include the JavaScript source files

Add the required JavaScript source files to your HTML page. The Kinect for Windows SDK JavaScript APIs are defined in Kinect-1.8.0.js. HTML5 Kinect UI elements such as cursors and buttons require the JQuery API.

<script type="text/javascript" src="Kinect-1.8.0.js"/>
<script type="text/javascript" src="https://ajax.aspnetcdn.com/ajax/jquery/jquery-1.9.1.min.js"/>

2. Initialize the Kinect sensor (optional)

When your HTML document is ready, call the connect function to connect to the Kinect web server. Note that this step is handled automatically if you use the default path and port.

Kinect.connect("https://localhost", 8181);

3. Retrieve an instance of the KinectSensor interface

The KinectSensor interface configures the sensor and sets handlers for events and frames.

var sensor = Kinect.sensor(Kinect.DEFAULT_SENSOR_NAME /*optional*/, 
                           function( sensorToConfigure, isConnected) /*optional*/ {...});

To support multiple sensors, use Kinect Explorer to identify the DeviceConnectionId for the sensors. Associate the name with the corresponding KinectSensorChooser to the SensorChooserMap in the web server.

var specificSensorChooser = new KinectSensorChooser();
specificSensorChooser.RequiredConnectionId = "USB\VID_045E&PID_02C2\5&2080E735&0&6";
var defaultSensorChooser = new KinectSensorChooser();
var webserver = new KinectWebserver();
webserver.SensorChooserMap.Add("specific", specificSensorChooser);
webserver.SensorChooserMap.Add("default", defaultSensorChooser);
specificSensorChooser.Start();
defaultSensorChooser.Start();
webserver.Start();

On the web client, use the assigned names to select the sensor to use.

var defaultSensor = Kinect.sensor(Kinect.DEFAULT_SENSOR_NAME);
var specificSensor = Kinect.sensor("specific");

4. Enable and configure the Kinect sensor streams

Kinect sensor streams are enabled and configured using JSON. You can retrieve the current configuration by calling the getConfig function.

sensor.getConfig( function (configData) { ... });

The following example demonstrates how to use the postConfig function to enable and configure the Kinect sensor streams. All valid stream names are shown in this example.

var configuration = {
 
    "interaction" : {
        "enabled": true,
    },
 
    "userviewer" : {
        "enabled": true,
        "resolution": "640x480", //320x240, 160x120, 128x96, 80x60
        "userColors": { "engaged": 0xffffffff, "tracked": 0xffffffff },
        "defaultUserColor": 0xffffffff, //RGBA
    },
 
    "backgroundRemoval" : {
        "enabled": true,
        "resolution": "640x480", //1280x960
    },
 
    "skeleton" : {
        "enabled": true,
    },
 
    "sensorStatus" : {
        "enabled": true,
    }
 
};
  
sensorToConfigure.postConfig( configuration );

5. Add a handler to process incoming stream frames

To process incoming data streams from the sensor, call the addStreamFrameHandler function to set a callback function that is called whenever a data frame is received.

sensor.addStreamFrameHandler( function(frame) {
                
    switch (frame.stream) {
        case Kinect.SKELETON_STREAM_NAME:
            for (var iSkeleton = 0; iSkeleton < frame.skeletons.length; ++iSkeleton) {
                var skeleton = frame.skeletons[iSkeleton];
                            
                skeleton.trackingId;
                skeleton.trackingState;
                skeleton.position;
                            
                for (var iJoint = 0; iJoint < skeleton.joints.length; ++iJoint) {
                    var joint = skeleton.joints[iJoint];
                    joint.jointType;
                    joint.trackingState;
                    joint.position; 
                }
            }
            
        break;
    }
});

6. Hook up the user interaction stream

To allow users to interact with your application, call the KinectUI.createAdapter function to create a KinectUIAdapter interface. This interface allows users to use a hand cursor to interact with Kinect-enabled controls.

// Create a UI adapter for the hand cursor and Kinect buttons.
var uiAdapter = KinectUI.createAdapter(sensor);

// Hook up DOM elements that are annotated with the "kinect-button" class to allow them to be pressed by Kinect interactions.
uiAdapter.promoteButtons();

// Create a HandPointerCursor interface that can display or hide the user's hand cursor.
var cursor = uiAdapter.createDefaultCursor();

Users can interact only with DOM elements that are annotated with the "kinect-button" class, as shown in the following HTML.

<link rel="stylesheet" type="text/css" href="kinect-1.8.0.css" />

<div class="kinect-button">
    <span>Test</span>
</div>

7. Display real-time bitmap streams

To display real-time bitmap streams from the sensor, create a canvas as shown in the following HTML.

<canvas id="userViewerCanvas" class="streamImageCanvas" width="640" height="480" />

Connect the bitmap streams to the canvas by calling the KinectUIAdapter.bindStreamToCanvas function.

var userViewerCanvasElement = document.getElementById("userViewerCanvas");
uiAdapter.bindStreamToCanvas(Kinect.USERVIEWER_STREAM_NAME, userViewerCanvasElement, 640, 480)

8. Add a handler to respond to events from the sensor (optional)

For troubleshooting and interaction purposes, it is often useful to monitor the events that are fired by the Kinect sensor. The following example demonstrates how to call the addEventHandler function to set a callback function that is called whenever an event is received. All valid events are shown in this example.

sensor.addEventHandler( function(event) {
 
    switch (event.category) {
 
        // Respond to user state changes.
        case Kinect.USERSTATE_EVENT_CATEGORY:
            switch (event.eventType ) {
 
                // This event signals a change to who the primary interacting user is.
                case Kinect.PRIMARYUSERCHANGED_EVENT_TYPE:
 
                    event.oldValue
                    event.newValue // trackingId
                    break;
 
                // This event signals a change to the states of the tracked users.
                case Kinect.USERSTATECHANGED_EVENT_TYPE:
 
                    event.userStates[0].id
                    event.userStates[0].userState // This is either "tracked" or "engaged".
 
                    sensor.postConfig( {
                        "backgroundRemoval": {
                            "trackingId": /*Find the tracking ID of the engaged user./,
                        }
                    })                                
                    break;
            };
            break;
 
        // This event signals that the sensor is connecting or disconnecting.
        case Kinect.SENSORSTATUS_EVENT_CATEGORY:
            if ( !event.connected)
            {
 
            }
            break;
    };
});

9. Extend functionality with custom streams (optional)

The Kinect web server can be extended to supply additional streams to the HTML5 application. Due to bandwidth limitations, you should limit the number of real-time image streams that you create. Perform any image processing on the server side to avoid using JavaScript to perform bit-level manipulation. For an example of how to implement custom streams, refer to the implementation of the Microsoft.Kinect.Toolkit.BackgroundRemoval feature.

Implementing a custom stream requires the following steps:

  1. Create a new stream type that derives from SensorStreamHandlerBase.
  2. Override the ProcessDataStream virtual functions to get access to the raw Kinect sensor data.
  3. Implement the ISensorStreamHandlerFactory interface as a factory class.

The following example demonstrates how to implement a custom stream.

class MyCustomSensorStreamHandler : SensorStreamHandlerBase
{
    ...
}

class MyCustomSensorStreamHandlerFactory : ISensorStreamHandlerFactory
{
    public ISensorStreamHandler CreateHandler(SensorStreamHandlerContext ownerContext)
    {
        return new MyCustomSensorStreamHandler(ownerContext);
    }
}

KinectWebserver webserver = new KinectWebserver();
webserver.SensorStreamHandlerFactories.Add(new MyCustomSensorStreamHandlerFactory());

See Also

Kinect for Windows JavaScript Reference