Multipoint with the Kinect - Part I
Introduction
Kinect multipoint was a project intended to work with Kinect multipoint, mouse mischief, and other multi-mouse applications and some desktop touch software. It was intended to map Kinect user coordinates of multiple people to multiple mouse coordinates (x,y), move a mouse, or simulate multiple mouse clicking behavior. The project started out with vbscript, and then I progressed to working in vb.net and c++ to create a mouse emulator library for the project. I am currently working on a full vb.net solution but currently only have Kinect code available to share with everyone in vb.net. The mouse emulator code is not available yet and I will update the article whenever I get it working. Please bear with me below as the code may work better on your Kinect for windows or the newer Kinect for windows 2.0. This starts part I of II in my Kinect multipoint article session. Part II will be written once I finish source code for the mouse emulator.
I. Requirements
Current requirements
1. DOTNET framework 4.5 or higher
2. Kinect SDK 1.7 or higher
3. Kinect Multipoint project (http://kinectmultipoint.codeplex.com)
Future requirements
1. DSF runtime 1.0 from wdk 7.1 (need the one from wdk 7.1!) <- for the mouse emulation code (will post that section later once finished with it).
The full requirements for development are:
- WDK 7.1 sample code with header files
- The softhidreceiver.dll file
- Full Kinect SDK with source code
- Optional – download KinectContrib for easier development of Kinect projects.
II. Kinect Interaction with Users
First, I must describe how the Kinect interacts with users before proceeding into the code. This process involves an infrared beam that hits the user/s and returns raw data about each user back to the Kinect software. Furthermore, the Kinect software API then takes the data and creates a depth image and skeleton of it based on the position of the human in front of the Kinect. In addition, WPF (windows presentation foundation) is used to host the controls the Kinect needs to fire its events on the vb.net side. A picture box is used in WPF to capture the depth and skeleton parts used to represent a person. Skeleton parts are represented by dots that represent the center of mass on a person. For example, there are dots for the foot, head, hands, and many other skeleton parts. Think of rigging a 3D model with a skeleton as an example for the skeleton involved with the Kinect. Depth is the data representing what the Kinect detected whether humans or inanimate objects. It is used in most instances for identification of human users or players in a Kinect game. Sometimes it can be used for mapping inanimate objects and 3D reconstruction. Note: A human is usually color coded in a depth image based on his distance from the Kinect (a future sdk may change this but as far as I know this is true). A good tip to remember: if the player is black in the depth stream they are too far away for the Kinect to recognize. If the player is white they are too near the sensor. Please refer to figure 2 for a little picture of the color range to figure out about how far a user is from the Kinect. In the picture, the Default range is for the Xbox Kinect and near range is only for Kinect for windows (KFW). Let's dive into some code now. The code in figure 1 shows how my Kinect multipoint application is initialized with certain settings (color camera resolution of 640*480, depth resolution of 320*240, depth enabled, skeleton stream, and color stream enabled, and skeleton is set for smoothing to allow better precision on mouse coordinates). As you can see it looks like it is adding smoothing or precision to the initialization routine and different resolutions for Color and depth stream.
Figure1 – Kinect initialization routine
Private Sub kinectSensorChooser_KinectSensorChanged(ByVal sender As Object, ByVal e As DependencyPropertyChangedEventArgs)
Dim old As KinectSensor = CType(e.OldValue, KinectSensor)
'If old Is Nothing Then
' Return
'End If
StopKinect(old)
sensor = CType(e.NewValue, KinectSensor)
If sensor Is Nothing Then
Return
End If
Dim parameters As New TransformSmoothParameters()
parameters.Smoothing = 0.7F
parameters.Correction = 0.3F
parameters.Prediction = 0.4F
parameters.JitterRadius = 1.0F
parameters.MaxDeviationRadius = 0.5F
sensor.SkeletonStream.Enable(parameters)
sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30)
sensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30)
AddHandler sensor.AllFramesReady, AddressOf sensor_AllFramesReady
Try
sensor.Start()
Catch e1 As System.IO.IOException
'another app is using Kinect
kinectSensorChooser.AppConflictOccurred()
Catch ex As Exception
MsgBox(ex.Message)
End Try
End Sub
Figure 2 - Kinect sensor depth stream distance colors (How far is a person away from the Kinect?).
http://i.msdn.microsoft.com/dynimg/IC568993.png
III. Kinect Player identification and coordinate location
I have to identify players by looping through them as the Kinect sdk does not recognize more than 2 players at a time. First, I recognize players by retrieving all people recognized from the skeleton stream and copy to a skeleton array. Refer to figure 3. Second, I assign a random player ID used to identify the player by the Kinect system. Kinect requires a random ID used for custom identification can you believe that? For instance, player 1 gets a number between 5 and 9, player 2 a number between 10 and 14, and so on. As you can see every player is assigned a random id number within a 4 number range depending on what player they are. I show some sample ids up to player 6 in a table in Figure 6 to show how the id process works for the players. This identification tells us who a specific player is so we can send input to that player. Figure 4 contains the player identification code. I combine the players identification Third, I retrieve the x and y coordinates of a player using the CursorX and CursorY variables and find out how many players the Kinect recognizes using activeCount. The activeCount variable stores the number of people during each loop that the Kinect recognizes. I use it to figure out how many people need to have coordinates sent to the emulated mouse devices created. Notice the scaledright and scaledleft variables? They are there to give a hand that moves better precision when moving their specific emulated mouse. In the original application that gave me inspiration for this project there was a button to switch between left and right handed people that’s why they the variables are there. Figure 5 was the place I retrieved the x,y coordinates of the hand and whether the gesture to left click was made. Please refer to notes for information about the limitations of the code, more information about the project, and hints about what will be in part II.
Figure 3 – retrieve players
Private Sub sensor_SkeletonFrameReady(ByVal e As AllFramesReadyEventArgs)
' Dim j As Integer
' Dim s(5) As Skeleton
Dim playerid(6) As Integer
If IsNothing(sensor) Then
Application. Exit ()
End If
' For Each k In Microsoft.Kinect.KinectSensor.KinectSensors
' k.SkeletonStream.AppChoosesSkeletons = True
' sensor.SkeletonStream.AppChoosesSkeletons = True
Using skeletonFrameData As SkeletonFrame = e.OpenSkeletonFrame
If skeletonFrameData Is Nothing Then
Exit Sub
End If
' sensor.SkeletonStream.AppChoosesSkeletons = True
Dim allSkeletons(skeletonFrameData.SkeletonArrayLength - 1) As Skeleton
skeletonFrameData.CopySkeletonDataTo(allSkeletons)
Figure 4 – Identifying players by their specific random ID range (still sensor_SkeletonFrameReady)
'identify specific players by ID
'ResetValues()
' Using frame = k.SkeletonStream.OpenNextFrame(200)
'If IsNothing(Frame) = False Then
' Dim s(Frame.SkeletonArrayLength - 1) As Skeleton
' Frame.CopySkeletonDataTo(s)
'player tracking id initialization
' For j = 0 To 5
'allSkeletons.TrackingId = 0
'Next
'player #1
playerid(0) = CInt(Rnd() * 4)
' playerid(0) = 1
allSkeletons(0).TrackingId = playerid(0)
Log("player#1 tracking: " + allSkeletons(0).TrackingId.ToString)
'player #2
playerid(1) = CInt(Rnd() * 4 + 5) ' 5 - 9
'playerid(1) = 2
allSkeletons(1).TrackingId = playerid(1)
'player #3
' playerid(2) = 3
playerid(2) = CInt(Rnd() * 4 + 10) ' 10 - 14
allSkeletons(2).TrackingId = playerid(2)
'player #4
playerid(3) = CInt(Rnd() * 4 + 15) ' 15 - 19
allSkeletons(3).TrackingId = playerid(3)
'player #5
playerid(4) = CInt(Rnd() * 4 + 20) ' 20 - 24
allSkeletons(4).TrackingId = playerid(4)
'player #6
playerid(5) = CInt(Rnd() * 4 + 25) ' 25 - 29
allSkeletons(5).TrackingId = playerid(5)
'force each player to be moved to first blank spot
Dim tempList As New List(Of Skeleton)(allSkeletons)
tempList.RemoveAll(Function(sk) IsNothing(sk))
allSkeletons = tempList.ToArray
Log("Tracking id for player#1: " + allSkeletons(0).TrackingId.ToString)
Log("Tracking id for player#2: " + allSkeletons(1).TrackingId.ToString)
Log("Tracking id for player#3: " + allSkeletons(2).TrackingId.ToString)
'MsgBox(allSkeletons.Length)
'If allSkeletons.Length < 6 Then
' For j = 0 To allSkeletons.Length - 1
' ReDim allSkeletons(5)
' Next j
'End If
' For j = 0 To 5
** **Figure 5 – retrieve x,y coordinates,do a left click, and drop players not being used. (still sensor_SkeletonFrameReady)
For Each s As Skeleton In allSkeletons
' Dim d As Integer
Log("person#: " + i.ToString + " trackingstate: " + allSkeletons(i).TrackingState.ToString)
' Dim s(5) As Skeleton
's(j) = allSkeletons(j)
If IsNothing(s) Then
Exit Sub
End If
If s.TrackingState = SkeletonTrackingState.Tracked Then
activeCount = activeCount + 1
End If
If s.TrackingState = SkeletonTrackingState.PositionOnly Then
passiveCount = passiveCount + 1
End If
If s.TrackingState = SkeletonTrackingState.NotTracked Then
nottracked = nottracked + 1
End If
totalplayers = activeCount + passiveCount + nottracked
'Log("passive count: " + passiveCount.ToString + " date: " + Now.ToString)
If s.TrackingId > 0 Then
'sensor.SkeletonStream.ChooseSkeletons(s.TrackingId)
End If
'Select Case s.TrackingId
' Case Is = 1
' sensor.SkeletonStream.ChooseSkeletons(1)
' Case Is = 2
' sensor.SkeletonStream.ChooseSkeletons(2)
' Case Is = 3
' sensor.SkeletonStream.ChooseSkeletons(3)
' Case Is = 4
' sensor.SkeletonStream.ChooseSkeletons(4)
' Case Is = 5
' sensor.SkeletonStream.ChooseSkeletons(5)
' Case Is = 6
' sensor.SkeletonStream.ChooseSkeletons(6)
'End Select
' the first found/tracked skeleton moves the mouse cursor
If s.TrackingState = SkeletonTrackingState.Tracked Then
' make sure both hands are tracked
'If Skeleton.Joints(JointType.HandLeft).TrackingState = JointTrackingState.Tracked AndAlso Skeleton.Joints(JointType.HandRight).TrackingState = JointTrackingState.Tracked Then
Dim cursorX, cursorY As Integer
' get the left and right hand Joints
Dim jointRight As Joint = s.Joints(JointType.HandRight)
Dim jointLeft As Joint = s.Joints(JointType.HandLeft)
' scale those Joints to the primary screen width and height
Dim scaledRight As Joint = jointRight.ScaleTo(CInt(Fix(SystemParameters.PrimaryScreenWidth)), CInt(Fix(SystemParameters.PrimaryScreenHeight)), SkeletonMaxX, SkeletonMaxY)
Dim scaledLeft As Joint = jointLeft.ScaleTo(CInt(Fix(SystemParameters.PrimaryScreenWidth)), CInt(Fix(SystemParameters.PrimaryScreenHeight)), SkeletonMaxX, SkeletonMaxY)
' relativemouselocation.Content = jointRight.Position
' figure out the cursor position based on left/right handedness
If LeftHand.IsChecked.GetValueOrDefault() Then
cursorX = CInt(Fix(scaledLeft.Position.X))
cursorY = CInt(Fix(scaledLeft.Position.Y))
Else
cursorX = CInt(Fix(scaledRight.Position.X))
cursorY = CInt(Fix(scaledRight.Position.Y))
End If
Dim leftClick As Boolean
' figure out whether the mouse button is down based on where the opposite hand is
If (LeftHand.IsChecked.GetValueOrDefault() AndAlso jointRight.Position.Y > ClickThreshold) OrElse ((Not LeftHand.IsChecked.GetValueOrDefault()) AndAlso jointLeft.Position.Y > ClickThreshold) Then
leftClick = True
' MsgBox("clicked")
Else
leftClick = False
End If
For g = 0 To 5
allSkeletons(g).TrackingId = 0
For f = 0 To activeCount - 1
Select Case f
Case Is = 0
playerid(0) = CInt(Rnd() * 4)
s.TrackingId = playerid(0)
Case Is = 1
playerid(1) = CInt(Rnd() * 4 + 5) ' 5 - 9
'playerid(1) = 2
s.TrackingId = playerid(1)
Case Is = 2
' playerid(2) = 3
playerid(2) = CInt(Rnd() * 4 + 10) ' 10 - 14
s.TrackingId = playerid(2)
Case Is = 3
playerid(3) = CInt(Rnd() * 4 + 15) ' 15 - 19
s.TrackingId = playerid(3)
Case Is = 4
playerid(4) = CInt(Rnd() * 4 + 20) ' 20 - 24
s.TrackingId = playerid(4)
Case Is = 5
playerid(5) = CInt(Rnd() * 4 + 25) ' 25 - 29
allSkeletons(5).TrackingId = playerid(5)
End Select
Next
Next
'if i is less then the total amount of players then send players coordinates for person that is active.
If i <= 5 Then
Select Case True
Case Is = s.TrackingId >= 1 And s.TrackingId <= 4
i = 0
playeractive(i) = True
player1xy.Content = cursorX & ", " & cursorY & ", " & leftClick
Status.Text = "player1 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
' sensor.SkeletonStream.ChooseSkeletons(1, 2)
Case Is = s.TrackingId >= 5 And s.TrackingId <= 9
i = 1
playeractive(i) = True
player2xy.Content = cursorX & ", " & cursorY & ", " & leftClick
Status.Text = "player2 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
Case Is = s.TrackingId >= 10 And s.TrackingId <= 14
i = 2
playeractive(i) = True
player3xy.Content = cursorX & ", " & cursorY & ", " & leftClick
Status.Text = "player3 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
Case Is = s.TrackingId >= 15 And s.TrackingId <= 19
i = 3
playeractive(i) = True
player4xy.Content = cursorX & ", " & cursorY & ", " & leftClick
Status.Text = "player4 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
Case Is = s.TrackingId >= 20 And s.TrackingId <= 24
i = 4
playeractive(i) = True
player5xy.Content = cursorX & ", " & cursorY & ", " & leftClick
Status.Text = "player5 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
Case Is = s.TrackingId >= 25 And s.TrackingId <= 29
i = 5
playeractive(i) = True
player6xy.Content = cursorX & ", " & cursorY & ", " & leftClick
Status.Text = "player6 identified" & cursorX.ToString & ", " & cursorY.ToString & ", " & leftClick.ToString
End Select
Log("person #: " + i.ToString + "Tracking ID: " + s.TrackingId.ToString)
currentperson.Content = i.ToString + "Tracking ID: " + s.TrackingId.ToString
End If
'If playeractive(i) = True And i >= 0 And frame.SkeletonArrayLength > 0 Then
'NativeMethods.SendMouseInput(cursorX, cursorY, CInt(Fix(SystemParameters.PrimaryScreenWidth)), CInt(Fix(SystemParameters.PrimaryScreenHeight)), leftClick, totalplayers, i)
'End If
'if total players is 1 or greater send coordinate data.
If totalplayers >= 1 Then
DefineMouseData(cursorX, cursorY, leftClick)
End If
'make the below code active when I get multiple player tracking working
If playeractive(i) = True Then
' Environment.SetEnvironmentVariable("Player" + i.ToString + "xcoords", player(i).bytex.ToString, EnvironmentVariableTarget.Machine)
' Environment.SetEnvironmentVariable("Player" + i.ToString + "ycoords", player(i).bytey.ToString, EnvironmentVariableTarget.Machine)
'Environment.SetEnvironmentVariable("Player" + i.ToString + "leftclick", player(i).leftclick.ToString, EnvironmentVariableTarget.Machine)
'System.Threading.Thread.Sleep(300)
End If
If i <= 5 Then
If playeractive(5) = True Then
'if 6th player exit for loop and open next frame.
playeractive(5) = False
Exit For
Else
'd = d + 1
End If
End If
End If
NumPassive.Content = "passive count: " + passiveCount.ToString
Numactive.Content = "active count: " + activeCount.ToString
nottrackedplayers.Content = "not tracked players: " + nottracked.ToString
playeractive(i) = False
' Next
For h = 0 To activeCount - 1
If h >= 2 Then
If activeCount - 1 > 0 Then
sensor.SkeletonStream.ChooseSkeletons(playerid(h), playerid(h + 1))
End If
End If
Next
Next
ResetValues()
End Using
'Next k
End Sub
Figure 6 - Table of Sample player identification numbers.
| Player # | Identification # |
| 1 | 5-9 |
| 2 | 10-14 |
| 3 | 15-19 |
| 4 | 20-24 |
| 5 | 25-29 |
| 6 | 30-34 |
IV. Notes for this project
Below is a list of extra information I may have forgot to add in this article and some hints for the code later for part II which is still in progress:
1. Both the Xbox Kinect and Kinect for windows recognize 6 players total per Kinect (excluding skeleton tracking for Xbox 360 Kinect hence the weird looping of players above) and the number of Kinects depends on CPU processing power. The new Kinect for Windows rec and Xbox One Kinect recognize and identify 6 players.
2. The mouse emulator code helper dll is almost finished. If someone wants to take a shot at it and get the whole Kinect multipoint project working efficiently they can. I only need to figure out the reason why the queue input function to softhidreceiver.dll does not take my data from vb.net or c++ side. It wants a com safe array of variants the data passed is correct but the format of the com data is not.
3. As it stands now you can change the mouse to any device you need to emulate with the Kinect but require you to have hardware and low level knowledge of the touch or mouse related device. Use Microsoft’s device simulation framework to do so. I’ll post an article on that side later when I finish it and link the two articles together in this one.
4. There is no right click functionality yet. That will happen after I finish the vb.net emulator dll helper code.
5. Kinect is not recommended on tablets as tablets might not have the proper processing power to handle many Kinect's (Mini-desktop towers are okay).
6. Ignore commented out code as chances are I will not use it in the future.
VI. Links for development with Kinect and Source Code
1. KinectContrib – http://kinectcontrib.codeplex.com (Optional) – These are some nice open source templates for coding with the Kinect sdk until official ones are replace by Microsoft.
2. Article Source code – http://kinectmultipoint.codeplex.com
3. Kinect Mouse project – http://kinectmouse.codeplex.com
4. WDK 7.1 (only for part II) – http://www.microsoft.com/en-us/download/details.aspx?id=11800.
5. Device simulation framework (latest version is obtained from WDK 7.1).
** ** 6. WDK 8 (Optional and for part II) – this is optional improvement to help with running through wdk and dsf source code from wdk 7.1 – http://go.microsoft.com/fwlink/p/?LinkID=324284.
7. Kinect SDK 8.0 – http://go.microsoft.com/fwlink/?LinkID=323588
8. Kinect Toolkit 8.0 - http://go.microsoft.com/fwlink/?LinkID=3235899
9. Human Interface Guidelines - these are the guidelines Microsoft made for development with the Kinect sdk.
Follow them and you will have nice applications that run well with the sdk-http://go.microsoft.com/fwlink/?LinkID=247735