Kinect for Windows Product Blog

Kinect for Windows news gets new address, keeps same great community

Kinect for Windows Team — Mon, 07 Nov 2016 21:24:14 GMT

For the past several years, this blog has brought you news about the latest Kinect for Windows technology and its innovative applications in medicine, education, manufacturing, retailing, performance art, and more. We’ve been awed by the creative ways in which the developer community has harnessed Kinect for Windows to build solutions that allow users to interact naturally with computing technology. Earlier this year, we reported that developers would soon be able to read Kinect’s RGB, IR, and depth data with the Windows 10 Anniversary Update and a Kinect driver update. (Kinect’s microphone data was already available to the Windows 10 UWP audio and speech recognition APIs). And, as explained in this Windows Developer blog post, we are pleased to announce the availability of these functionalities—along with the much-requested ability to access Kinect skeletal tracking data. You can find code samples at GitHub as part of the Windows universal samples, and you can download the latest Kinect driver from your computer’s Device Manager. With the Windows 10 APIs and the new driver, developers can incorporate Kinect functionality into Universal Windows Platform (UWP) applications. Moreover, these same APIs will handle rich data from other sensors, provided you have the appropriate driver. Now that these capabilities have been implemented in Windows 10, it only seems fitting to fold Kinect for Windows news into the Windows Developer blog, at https://blogs.windows.com/buildingapps/. Beginning today, that’s where you can keep abreast of the current developments in human-computer interactions—and how you can use Kinect for Windows functionality to create solutions that will run across the entire UWP ecosystem. As part of this effort to consolidate your sources for developer information, we are also merging Kinect for Windows social media with the Windows Developer Facebook and Twitter accounts. The Kinect for Windows Facebook and Twitter accounts will remain open until Nov 15th, 2016, but now would be a good time to start following the Windows Dev accounts for the latest info about developing UWP apps. And of course, now is also the ideal time to hop on over to our new home on the Windows Developer blog and add it to your favorites. And as always, we’re eager to hear from you, so please join the conversation at the Kinect for Windows v2 SDK forum. Happy developing!

The Kinect for Windows Team

Key links

Is your job a pain in the neck? Kinect might help.

Kinect for Windows Team — Wed, 14 Sep 2016 20:37:27 GMT

Do you sometimes have a stiff, aching neck after a day of staring at your computer? If so, you’re far from alone. According to an article published in The Journal of the Canadian Chiropractic Association, over a third of regular computer users report work-related neck pain (WRNP) annually. So, just like you’ve been telling your friends, your job really can be a pain in the neck. And grousing aside, WRNP has serious public health implications in terms of decreased productivity, lost wages, and payments for medical treatment. Back in 1997, the National Institute for Occupational Safety and Health pegged the cost of all work-related muscular disorders at $13 billion annually in the United States, a figure that later researchers revised upward to $45 to $54 billion. Granted, not all of those costs are the result of WRNP—but a lot of them are. With so much at stake, it would be great to have a simple, affordable way of evaluating workers’ posture in order to eliminate ergonomically awkward neck positions. The standard device used to study such postures is an electrogoniometer, a relatively expensive piece of specialized equipment that measures joint angles. This led researchers at Urmia University of Medical Sciences in Iran to explore the Microsoft Kinect sensor as a cost-effective alternative for identifying troublesome neck postures. Using the Kinect for Windows software development kit, they wrote an application that calculates a subject’s neck angle based on the Kinect sensor’s skeletal tracking data. They tested the results of their Kinect-based program against those from an electrogoniometer and found their program’s results to be reliable and accurate. Moreover, the Kinect-based measurement was quicker and required less equipment. They concluded that the Kinect for Windows system offers a promising, low-cost, easy-setup alternative for studying neck posture. So, think about that the next time you’re rubbing your neck after hours on the computer. Maybe a little time in front of the Kinect sensor could identify posture changes that would alleviate your suffering. You might still find your job a pain in the neck, but now it would be figurative rather than literal. If you’d like to share information about a Kinect for Windows v2 solution that you’ve developed, contact us. We may be able to feature your story in a future blog.

The Kinect for Windows Team

Key links

Keeping an eye on pugnacious pigs

Kinect for Windows Team — Thu, 11 Aug 2016 15:21:40 GMT

Notwithstanding their docile image in cartoons and literature (think Porky Pig from the Looney Tunes cartoons or Wilbur in the novel Charlotte’s Web), pigs can be surprisingly aggressive, as any hog farmer can tell you. Their belligerent behaviors become especially prominent when several unfamiliar pigs are penned together, as often happens after the pigs have been weaned. Being social animals, the pigs fight to establish a dominance hierarchy, butting and chasing one another to determine who gets the most food and space in confined quarters. These aggressive behaviors can be costly, as they result in injuries and, in the case of submissive pigs, a failure to thrive.

Mounted above the pig pens, the Kinect sensor captured infrared depth images of the pigs' behaviors.

And what, you may ask, does this have to do with Kinect for Windows? Well, researchers at Korea University in Sejong have devised an accurate, low-cost system that uses a single Kinect v2 sensor, mounted overhead, to monitor pig behavior in a commercial pigpen. During the study, the sensor’s depth images were fed into two binary-classifier support vector machines (SVMs), which recognized and classified aggressive behaviors with greater than 90 percent accuracy. Armed with this information, hog farmers could take action to curb the aggressive behaviors. The Kinect v2’s infrared-based depth sensing proved ideal for monitoring behavior in a commercial pigpen, where heat lamps and artificial lighting can interfere with the quality of images from a standard RGB camera. The sensor provided the researchers with a real-time depth map showing the pigs’ position frame by frame, providing the SVMs with ample data to detect and classify aggressive behaviors. The researchers touted both the accuracy of the system and its simplicity and low cost when compared to solutions that use stereo RGB cameras, and they plan to test the system more widely in commercial hog farms. If you’d like to share information about a Kinect for Windows v2 solution that you’ve developed, contact us. We may be able to feature your story in a future blog.

The Kinect for Windows Team

Key links

Learn about developing with Kinect for Windows
Download the Kinect for Windows SDK 2.0
Read the Korean research paper
Follow us on Facebook and Twitter

VNTANA HOLLAGRAM brings AR to Marketing

Kinect for Windows Team — Fri, 01 Jul 2016 08:06:14 GMT

You’ve built the world’s only interactive, scalable hologram system—so, what do you do for an encore? If you’re augmented reality (AR) company VNTANA, you link that hologram system to social media and give it data collection capabilities, creating HOLLAGRAM, a first-of-its-kind marketing tool. The HOLLAGRAM system consists of the Kinect v2 sensor, a powerful PC, a holographic display screen, and, of course, a load of innovative software. The entire setup is extremely compact, suitable for use in even fairly small retail locations. Users stand in front of VNTANA’s HOLLAGRAM, and their image is captured by the Kinect sensor’s body-tracking feature and its infrared sensor. The image is cut out from its background through VNTANA’s proprietary software, so there’s no need for a green screen or any wearable sensors. This image is then processed in real time through VNTANA’s hologram rendering engine, and, almost magically, users immediately see their own hologram live on the display. Taking advantage of Kinect for Window’s gesture control capabilities, users can interact directly with other holographic objects, such as a celebrity or product, that have been preloaded in the digital space. All they have to do is move their arms and hands in front of the display. Want to spin a corporate logo? Just move your hand in a circle. Want to be a virtual DJ? The video below shows you how easily it can be done with HOLLAGRAM. https://vimeo.com/129937744 VNTANA CEO and co-founder Ashley Crowder notes that Kinect for Windows was crucial to the development of HOLLAGRAM. "The Kinect was the first affordable IR sensor readily available on the market and it's the only sensor that can track the full body without wearables. Not having wearables is very important to VNTANA, as we're focused on creating group experiences in public spaces like malls, stadiums, trade shows, and resorts." As you can see in the accompanying video, users are captivated by the real-time experience of seeing themselves as an interactive hologram. And while giving people a good time is a worthy goal all on its own, it’s the social media and data collection aspects of HOLLAGRAM that have captured the attention of such major companies as Pepsi and Chinese drone manufacturer DJI. Immediately after the experience, users receive a branded email with a GIF, video, or photo of their holographic antics that they can share on social media. This social media component builds brand awareness. Meanwhile, HOLLAGRAM collects voluntary and passive user data, including email contacts and number of impressions, thereby building a database of potential customers. Microsoft Azure powers this aspect of the HOLLAGRAM system, compiling and categorizing the data seamlessly in the cloud. Commenting on Azure, Crowder praises it as an “affordable way to scale quickly. It enabled us to build out our social media integration in six days for a client.” Microsoft’s BizSpark program, which offers resources to promising startups, provided VNTANA with the Kinect v2 sensor (about a year before its general release) and access to Azure cloud services, along with tech support for getting the most out of the Kinect for Windows SDK and Microsoft Azure.

The Kinect for Windows Team

Key links

Bringing quality education to the countryside

Kinect for Windows Team — Fri, 24 Jun 2016 11:00:20 GMT

Providing top-notch schooling in rural communities is a problem around the world—and one that China, a country with more than 600 million rural residents, is committed to tackling. A big part of this effort centers on using technology to bring the urban classroom, with its greater teaching resources, to the rural schoolhouse. Among the Chinese technology companies that are working to help deliver quality education to rural locations, Beijing Trainsfer stands out. This independent software vendor has created a long-distance teaching solution based on two Microsoft technologies: Kinect for Windows and Skype. Unlike other long-distance video solutions, which suffer from inferior video tracking and require expensive audio transmission equipment, the Kinect/Skype application offers a low-cost, high-quality way for urban educators to interact naturally with students in remote locations. [embed]https://www.youtube.com/watch?v=ObeGtwRMPhE&feature=youtu.be[/embed] This subtitled video demonstrates the Kinect-enabled technology that is bringing urban teaching to rural classrooms. With a Kinect for Xbox One sensor in both the urban and rural classrooms, Trainsfer’s system employs a proprietary tracking robot that takes advantage of the Kinect sensor’s body tracking capabilities to accurately follow the movements of the teacher and the students. Meanwhile, the sensors’ RGB camera and microphone array faithfully capture the images and audio at each end. Then, by using a plug-in based on the DirectShow technology, the Kinect data is processed for remote transmission via Skype. With the Trainsfer solution in place, the distance between teacher and students becomes irrelevant. It’s as if everyone were located in the same classroom. Through this virtual classroom, the system provides a cost-effective way to bridge the educational divide between urban and rural settings. [caption id="attachment_2315" align="alignleft" width="457"] Students in this rural classroom are being taught remotely through the Trainsfer long-distance learning solution.[/caption] [embed]https://www.youtube.com/watch?v=IVV9uQgLz7I&feature=youtu.be[/embed] In this video, we see the installation of the Trainsfer long-distance teaching solution in two rural schools. The solution is now being tested in four classrooms, two each in Qinghai and Guizhou provinces, bringing quality education to about 200 rural students. If this trial is successful, the Kinect-enabled remote-learning solution could be adopted across China, bringing better educational opportunities to tens of millions of rural students.

The Kinect for Windows Team

Key links

Kinect-enabled VR holds productivity promise

Kinect for Windows Team — Thu, 09 Jun 2016 15:27:22 GMT

Ah, the old office directory: page after page, or screen after screen, of names and job titles. Wouldn’t it be great to have a virtual directory that displayed employees’ faces, included the company hierarchy, and grouped employees according to responsibilities and expertise? Better yet, what if that directory could be navigated by using natural hand gestures? By combining the power of the Microsoft Kinect sensor and the Oculus Rift virtual reality (VR) system, French IT consulting firm OCTO Technology has created just that—a VR company directory that lets users traverse the company’s human resources and find the right employee with a flip of their hand. OCTO developers Muriel Caron and Vincent Guigui used BabylonJS, an open-source 3D/VR web framework, to display a VR representation of the company directory in which employees are grouped by expertise, skill, or interest. The resulting display is a floating image of the company’s workforce, as shown in the video below. The head-tracking capabilities of the Oculus Rift follows users as they look in any direction of the display, while the body-tracking technology of Kinect for Windows allows users to grab or release employee entries by moving their hand forward or backward. https://www.youtube.com/watch?v=GTIWw-3uACo “Mixing technologies which were not made to work together was a bit of a challenge,” says Guigui. “We had to devise solutions to properly track user movements using Kinect, feed them into a VR environment created using web technology (everything is running inside the Internet Explorer web browser), then display it inside the Oculus Rift headset. All with a very low latency, to reduce the motion sickness often encountered during VR experiences.” Challenging to be sure, but the OCTO team succeeded in creating an immersive world that feels real and where the movement and position of objects seem consistent with the user’s point of view and actions. Moreover, Guigui sees this as just one example of the many ways that we can use VR in productivity applications. “VR will unleash productivity,” he predicts, “freeing users from the limitations of common displays and interfaces, letting them focus solely on the information and the objects they want to interact with. Think of an employee having any number of virtual screens floating above her desk, or even following her while she moves from one room to another. Those virtual screens could be any size or shape and have dynamic opacity. OCTO is already working on these future immersive experiences, using mixed-reality devices like HoloLens to break or bridge the boundaries between realities, allowing physical and digital entities to coexist and interact.” Heady stuff, and we’re pleased that Kinect for Windows will be there to help those users navigate their way—naturally and intuitively—through this abundance of information.

The Kinect for Windows Team

Key links

Pumping iron with Kinect for Windows

Kinect for Windows Team — Thu, 02 Jun 2016 18:06:21 GMT

No, you aren’t going to clean and jerk the Kinect sensor—and given its compact size, that wouldn’t be much of an exercise anyway. What you can do, with the tutorial provided by Vangos Pterneas (a Microsoft Kinect Most Valuable Professional), is build a Kinect for Windows application that will recognize your barbell and measure how far you’ve raised it from the floor. [embed]https://www.youtube.com/watch?v=R0LygrFHoEE[/embed] While this application can be useful for weightlifters, letting them track their progress day after day, the real beauty of Pterneas’s tutorial is that it shows developers how to use the Kinect for Xbox One sensor to recognize and track a physical object—in this case, a barbell. As Pterneas notes, “Kinect can recognize human body joints out-of-the-box. Body tracking is built into the SDK. However, there is no built-in way to recognize physical objects.” Rather than rely on tracking objects by processing RGB color images, Pterneas uses the Kinect sensor’s depth detecting capabilities to create a 512 x 424 point cloud, complete with the horizontal distance between each point and the sensor plane. He then uses the sensor’s body tracking capabilities and the Kinect for Windows SDK’s BodyIndexFrames feature to identify the weightlifter’s hand positions in three-dimensional space, and from those data he can determine the depth points that make up the barbell. After a few more refinements—like excluding depth points that belong to the weightlifter’s body—Pterneas’ app processes the barbell points to calculate the distance, height, and angle of the bar. Want to see the code? It’s on GitHub, so go take a look. And while you work your way through the tutorial, keep in mind how you could adapt this approach to create other applications that measure the position of physical objects as they are moved by Kinect-recognized body parts.

The Kinect for Windows Team

Key links

Getting your kicks with Kinect for Windows

Kinect for Windows Team — Wed, 18 May 2016 16:00:05 GMT

A wall, a ball, and a kid: think about it. When you were a youngster on a playground, and a wall and a ball were at hand, what would you do? If you were like most kids, you’d kick or throw that ball against the wall, moving around with the unbridled energy of youth as the ball bounced back to you, again and again. It’s exactly that sort of natural, simple exercise that Wall Ball, a Kinect for Windows game, seeks to recreate. Or as psychologist and game developer Tino Ågren of Mixxus Studio says, “A kid in the schoolyard moves around without thinking about it being ‘exercise.’ I like games to work in the same way—you turn Wall Ball on and start moving around, getting your pulse up, just because it’s fun.” [embed]https://www.youtube.com/watch?v=qFTv-q2cccg&feature=youtu.be[/embed] With Wall Ball, you don’t need a playground or a physical wall or ball—just a Kinect for Xbox One sensor and a Kinect Adapter for Windows, which allows the sensor to be hooked up to a compatible Windows PC. You can download the game itself from the Windows Store. During game play, the Kinect sensor’s skeletal tracking follows your movements as you simulate kicking an onscreen soccer ball (you get five balls in each round). Every time one of your kicks hits the wall, you score a point—and you can earn bonus points when you hit objects that randomly appear on the wall. You also need to watch the ball as it rebounds from the wall—if you let it get past you, you’ll be penalized one ball. The game offers three levels of play: easy (or beginner), standard, and seated mode (for use by people with mobility issues). In creating Wall Ball, Ågren used the Kinect Unity package in the Kinect for Windows SDK 2.0, which allowed him to develop the entire game in Unity. The physics simulations, which enable the skeletal tracking to accurately predict the flight of the ball, were the biggest challenge. Once he had the physics down, he knew he was on the way to creating an enjoyable way to burn off some physical energy anytime you have a few minutes. Ågren is especially happy to have made the game available through the Windows Store, which he feels is “a really good platform…a good way to reach a lot of different people.” He hopes to see Wall Ball and Country Ramble, another of Mixxus Studio’s Kinect for Windows games, reach beyond typical video gamers. He even foresees their use in retirement homes, helping elderly folks stay physically active by playing fun games that don’t require a mastery of electronic controllers. A blessing also for those of us who are still trying to figure out our TV universal remotes!

The Kinect for Windows Team

Key links

Kinect and UWP: new options in the Windows 10 Anniversary Update

Kinect for Windows Team — Fri, 13 May 2016 12:00:57 GMT

UWP apps today cannot access most of the data from a Kinect sensor. This post provides an overview of what we plan to deliver in the near future to begin bringing Kinect for Windows into the Universal Windows Platform (UWP) world. With the Windows 10 Anniversary Update and a Kinect driver update, Kinect developers will be able to read RGB, infrared (IR), and depth data from the Kinect sensor. The Kinect sensor’s microphone array is already visible to the UWP audio and speech recognition APIs today (although not all features from the Kinect for Windows SDK are accessible to UWP APIs)—this remains unchanged. Skeleton data is important to many Kinect apps. Through a supplemental SDK that will be available later this year, UWP developers will have access to skeletal poses. With the new capabilities in the Windows 10 Anniversary Update, developers can also use the existing Windows.Media.FaceAnalysis APIs with Kinect to detect and track faces. Existing Kinect Win32 apps and SDK capabilities will not be affected by this new UWP support.

Details: how it works

Today, there are several sensors that can provide rich correlated data, such as RGB, IR, or depth information. To make such data available to app developers in a device-independent manner, we introduced the Windows.Media.Capture.Frames APIs, a set of extensions to Media Capture that adds frame-by-frame access to RGB, IR, and depth data, as well as sensor correlation to the traditional image and video capture features, all with a consistent, familiar programming model. In addition, this extended model can give UWP apps access to custom streams from a sensor. In the case of Kinect for Windows, we plan to make use of the custom streams capability to deliver skeleton data from Kinect to UWP apps. This will require a small supplemental Kinect-specific SDK to decode the custom streams on the app side, which we’ll make available later this year. In order to make use of the new frame-by-frame capabilities of Media Capture, developers need a device capable of delivering rich data streams, as well as an updated driver. A driver update for Kinect will be available later this spring. By using the proven Media Foundation driver architecture, independent hardware vendors (IHVs) will be able to readily connect their own devices—including RGB, IR, depth, and other custom sensors—to the same Media Foundation infrastructure that now powers Kinect. The new API, together with the extensible driver model, will empower developers to target Kinect and supported third-party sensors in a hardware-agnostic way to gain better understanding of the world around them and the people in it. In summary, these are the elements that enable UWP apps to access Kinect:

The new Windows.Media.Capture.Frames extensions to Media Capture are available in the Windows 10 SDK preview, build 14332 or newer. These API extensions are device independent. To access IR and depth data, a compatible device and a matching device driver are also required.
A matching driver update for Kinect will be available later this spring.
To access skeleton data from Kinect, which is delivered through Windows.Media.Capture as a custom stream, a Kinect-specific supplemental SDK is needed, which we plan to release in the second half of 2016.

Join the Windows Insider Program so that you can access Windows 10 build 14332 and its preview SDK. Watch the Channel 9 video for a brief preview of the full capabilities that will be available when the Windows 10 Anniversary SDK and the supplemental Kinect SDK are released to the public, including full skeleton support. The video also tours the code for a simple demo app that demonstrates how to get frames from Kinect or any other RGB camera on your system.

The Kinect for Windows Team

Key links

Remix the music with Kinect

Kinect for Windows Team — Wed, 04 May 2016 16:42:13 GMT

This spring, visitors to Seattle radio station KEXP’s new home are being treated to the ultimate air-guitar experience: as they pass through a 4,000-square-foot art installation, their movements alter the music of some of the station’s most popular artists. This innovative project—dubbed a “virtual remix”—uses Kinect for Windows to capture the visitors’ body movements. The audio output, as well as the visual accompaniments to the music, both respond to visitors’ movements in an interactive and ever-evolving environment. As people move through the installation, they continuously reshape and customize the music of alt-J, Floating Points, Ólafur Arnalds, and Samaris, blurring the line between creator and audience. [embed]https://www.youtube.com/watch?v=I8h_CUOvuEQ[/embed] The installation is powered by four Kinect for Xbox One sensors mounted behind highly flexible mesh curtains hung in the corners of the installation space. Each sensor links to its own PC running Windows 8.1. The mesh curtains appear opaque to the Kinect sensors’ depth sensing technology but invisible to the sensors’ color camera and infrared emitter. The fabric is also pleasant to the touch, which encourages visitors to push on it. As they do so, the Kinect depth cameras collect data on how far and in what direction (left, right, up, or down) the mesh is deformed. These data are then translated into alterations to the music in real time. The music tracks had been previously split into various stems: the drum stem, the bassline stem, the melody stem, and the vocal stem. The Kinect depth data alters one or more stems, depending on how hard and in what direction a visitor pushes on the mesh fabric. These acoustical alterations are complex. A visitor’s push might change both the volume and the components of the drum stem; for example, making the high hat drum component more pronounced. The depth data also trigger changes in the ambient lighting, which is controlled by TouchDesigner. With multiple visitors pushing on different parts of the mesh curtains, the experience becomes even more complex, as the actions of each visitor alters the musical components and the accompanying lighting. And since the fabric is highly elastic, a visitor can push his or her entire body into a curtain, creating a large deformity in its shape. In fact, groups of visitors can push together into the fabric, generating huge deformities. The developers used the size of deformities to drive a lot of the musical interactions, reasoning that large deformities indicate a high level of enthusiasm or playfulness on the part of visitors. Given the large size of the installation, the developers had to set it up and test it in pieces, in a limited amount of time. Kinect Studio made this feasible, as it enabled the developers to record each session and then debug the coding later, outside of the live test environment. The developers also made heavy use of Kinect Common Bridge, an open-source application that let them work more quickly with Kinect for Windows. “Inside the Music of KEXP: an Interactive Music Experience” is part of Microsoft’s Music x Technology program, which explores emerging modes of artistic expression and celebrates advances in interactive technology. The KEXP installation premiered on April 16, the grand opening of the station’s new home, and runs through May 8. So if you’re in Seattle in the next few days and want to get out of the rain, stop on by and join the remix.

The Kinect for Windows Team

Key links