November 2018
Volume 33 Number 11
Machine Learning - Analyzing Olympic Sports Combining Sensors and Vision AI
By Kevin Ashley | November 2018
Internet of Things (IoT) and Machine Learning (ML) have been used for everything from inventory tracking and manufacturing quality control to traffic management and weather forecasting. One area where these two disciplines have come together to herald big changes is the arena of high-stakes sports. In the April issue of MSDN Magazine (msdn.com/magazine/mt846466), we explored how sensors are being used to track and improve performance in Winter Olympic sports events like alpine skiing. Now in this second installment, the focus shifts to the Summer Olympics, where IoT and ML are being leveraged to improve springboard diving.
It’s an area of urgency for the U.S. Olympic Team. There are four diving events—3-meter springboard, 10-meter platform, 3-meter synchronized diving and 10-meter synchronized diving—that account for 24 total available medals in each Olympics. Sensors and ML promise to improve training regimes and maximize athlete performance on the biggest stage.
Today, video is the predominant method of analysis in diving, as it has been for years. Coaches use slow motion and frame-by-frame video to analyze dives and give immediate feedback. However, this qualitative approach does not allow rigorous analysis and effective comparison of performance changes. To truly measure and document changes, quantitative analysis that marries video with data captured from specialized sensors is needed.
In this project, we use video and sensor technology to measure the diving takeoff—the critical first moments that are key to a successful dive. We designed a sensor to be placed underneath the tip of the springboard. It includes an inertial measurement unit (IMU), an accelerometer and a gyroscope to measure the important characteristics of the springboard takeoff. This includes the approach to the end of the board, the hurdle onto the end of the board, the pressing down and flexing of the springboard, and the lift into the air by the springboard. A great takeoff is critical for a great dive, with maximum height and spin needed for incredibly difficult dives like a forward 4 1/2 somersault.
Sensor data can determine if the board was maximally depressed on the way down, and if the diver rode the board all the way back up and waited for it to throw him or her into the air. Many divers don’t do this well, and tend to “skip off the board” on the way up, losing height and rotation in the process. This project aims to build a system that gives divers and coaches the information they need to rapidly improve the quality of the takeoff, allowing difficult and more difficult dives to be performed successfully.
Inside the Sensor
Attached to the underside tip of the diving board, the sensor is small and light enough to not affect the performance of the board, and operates for several hours on a single charge (Figure 1). The sensor is then ready to measure the angles, accelerations and timing during a diver’s takeoff. The most important parameter is maximum flexion angle, which determines how much energy the diver has transmitted into the board prior to takeoff.
.jpg "SensorKit Simulation and Springboard Setup")
Figure 1 SensorKit Simulation and Springboard Setup
A partial list of typical parameters that can be measured and are important to the coach include:
- Maximum downward flexion of the board; indicated by maximum board flexion.
- Hurdle flight time, which indicates maximum hurdle height attained.
- Board contact time, which measures the time from hurdle landing to takeoff.
- Velocity at board contact after hurdle, estimated from the time in the air.
- Velocity at takeoff, calculated from the angular velocity of the board tip and its acceleration.
- Maximum sideway tilt angle of the board tip, which determines if the takeoff occurred from one side of the board.
- Acceleration profile, which provides the raw data for more detailed analysis.
In the end-to-end solution, which is beyond the scope of this research article, this sensor data will be synchronized with video captured of the dive to a smartphone or tablet and then uploaded to the cloud. An intuitive UI will give coaches and athletes immediate access to the video, with controls for slow motion, step-forward and step-backward functions. The integration of sensor telemetry with video gives coaches and divers powerful insight into the mechanics of each dive, so they can understand what needs to be improved. In this solution we will only be calculating maximum flexion angle, maximum springboard velocity and maximum springboard acceleration.
Sensor Kit and the Diving App
The data is collected on our SensorKit sensor. The accelerometer and gyroscope provide the data to calculate flexion angle, hurdle flight and dive data, and transmit it wirelessly to the mobile app where the coach can see it. The app can trigger video recording based on sensor detection of a hurdle, and then stop recording after a set amount of time has passed after the sensor-detected takeoff. The app automatically uploads the captured data to the cloud using the SensorKit SDK, and provides a UI for coaches to tag and comment on athletes and their dives.
The sensor needs to be smart enough to process signal patterns specific to diving, which we narrowed to detecting hurdle steps from springboard oscillations, estimating takeoff time and sending calculated values back to the app (Figure 2). You can find the source code for the SensorKit SDK for diving apps at bit.ly/2ImaNQT.
.png "Physics of the Springboard Explained")
Figure 2 Physics of the Springboard Explained
SensorKit SDK provides data structures for the hurdle and the dive. The two classes are depicted in the code shown in Figure 3.
Figure 3 SensorDiveData and SensorHurdleData Classes
public class SensorDiveData
{
public long t { get; set; } // Timestamp
public double dt { get; set; } // Duration
public double count { get; set; } // Hurdle count
public double g { get; set; } // Acceleration, m/s^2
public double w { get; set; } // Angular velocity, dps
public double angle { get; set; } // Max flexion angle, deg
}
public class SensorHurdleData
{
public long t { get; set; } // Timestamp
public double dt { get; set; } // Duration
public double angle { get; set; } // Flexion angle, deg
public double g { get; set; } // Acceleration
public double w { get; set; } // Angular velocity, dps
}
This data is transmitted to the app through the SensorKit SDK, where it can be stored, for example using Cosmos DB. SensorKit provides a helpful class to handle Cosmos DB data.
Physics of the Dive
Our sensors work in two modes: raw and calculated. We’ll work with raw data from the sensors, both from our simulated springboard and from actual dives. There are several types of dives. For this article, we’ll focus on dives that start in a standing position at the back of the board, followed by a walk forward and a hurdle on to the end of the board, pressing it down to initiate a takeoff into the air. From our GitHub repository at github.com/kevinash/MSDNDivingSensor, you can use the diving.R file to load and parse sensor data, and calculate the values we used in the analytics section of this article.
In terms of the physics involved, consider pushing against something heavy coming toward you. The force you exert slows the object down, stops it and eventually accelerates it in the other direction. All this impacts the way we must look at finding the diving takeoff events from the velocity and acceleration curves. The problem is to determine where we are in the takeoff based on angular velocity or linear acceleration curves. A chart of board displacement looks like an oscillation graph: From the horizontal position the board goes down to its max flexion point, then goes back via horizontal 0 position and oscillates until it’s still.
To calculate max flex displacement, we can use angular velocity of the tip of the board or linear acceleration. Angular velocity will be out of phase with corresponding acceleration, however, because acceleration is the time derivative of velocity. The same relationship exists between displacement and velocity, because velocity is the time derivative of displacement. That means that velocity will be 0 at the max flexion (bottom point of displacement) of the diving board and acceleration is at max (Figure 2, Step 5). After that, the board goes to horizontal position where displacement equals 0 and velocity is max (Figure 2, Step 7). We calculated max flexion displacement angle by integrating angular velocity over time from our first point of zero (Figure 2, Step 5) to our second point of the takeoff where angular velocity becomes max (Figure 2, Step 7).
To start, in the R code (Figure 4) you can call the do-it-all function that parses files and displays results, like so:
CalculateSpringBoardDivingValues (sourceFile,lag, threshold, influence,1)
.png "Sensor Data Analysis in R")
Figure 4 Sensor Data Analysis in R
Let’s dig into the details of what’s happening inside this call. First, the function loads data from the raw sensor file source into the data frame, with this code:
df <- ReadNewFormatFile(dat2, debug_print = debug_print)
Sensors like gyro and accelerometer use somewhat-specific data formats, mostly for data optimization purposes. For example, the gyroscope produces angular velocity values in millidegrees per second (mdps). This is done to avoid using floating-point numbers. In our calculations we use the International System of Units (SI), used throughout most physics calculations. For acceleration, we also convert raw sensor values to m/s^2, a standard unit for acceleration. Here are the calculations:
# wx, wy, wz * 0.001 * degrees/second (ang. Velocity)
# ax, at, az = * 0.001 * 9.81 = m/s^2 (acceleration)
df$wx <- df$wx * 0.001 # convert to degrees per second
df$wy <- df$wy * 0.001 # convert to degrees per second
df$wz <- df$wz * 0.001 # convert to degrees per second
df$ax <- df$ax * 0.001 * 9.81
df$ay <- df$ay * 0.001 * 9.81
df$az <- df$az * 0.001 * 9.81
The direction of the angular velocity vector points perpendicular to the plane of the motion, while rotation happens around the axis of rotation, as shown in Figure 5. We can determine the axis of rotation by the axis with the most significant value change. A little math here, for angular velocity is the rate of change of the angle over time:
.png "Gyroscope-Sensing Angular Orientation")
Figure 5 Gyroscope-Sensing Angular Orientation
To obtain the angle, we can simply integrate angular velocity over time:
Our drift, or error, is minimized because we only look at the moment when the springboard starts bouncing. In R, this is the code for our integration:
# Interested only in [posStart:posEnd]
# When displacement is at MAX, velocity passes 0 point
revFirstMin <- which.min(rev(wSm$avgFilter[posStart:posEnd]))
# After been max negative
posLastMin <- posEnd - revFirstMin +1
# Next index when Angular Velocity goes
# through 0 is our starting integration point
nextZero <- min(which(floor(wSm$avgFilter[posLastMin:posEnd])>0))
# 0 position relative to Last Minimum
nextZero <- nextZero + posLastMin -1
# Max position after 0 position
nextMaxAfterZero <- which.max(wSm$avgFilter[nextZero:posEnd])
nextMaxAfterZero <- nextMaxAfterZero + nextZero -1
We also provided plotting functions to draw the results, as shown here:
PlotSingleChart(wSm$avgFilter[posStart:posEnd],
"Angular Velocity Smooth OUR Segment ","cyan4", kitSensorFile,
"Wx smooth deg.",TRUE, FALSE, d$Start[posStart:posEnd])
In the charts on Figure 6, we plot data from the gyro sensor while the board is still, through the hurdle and after the takeoff moment. It’s clear that the board reaches its maximum amplitude before the takeoff, after which the oscillation diminishes with time. To make our calculation more precise, and minimize drift error from the integration, we only use the moment between when the springboard starts oscillating to the moment when oscillation reaches its maximum.
.png "Charts for Flexion Angle During the Dive")
Figure 6 Charts for Flexion Angle During the Dive
Finally, we calculate the maximum flexion angle and the takeoff time, with this code:
timeTakeOff <- d$Start[posHorizontal]
timeStart <- d$Start[posLastMin]
writeLines (paste (
"Hurdle landing time (Start) = ", timeStart,"\n",
"Take Off time (End) = ", timeTakeOff,"\n",
"Board contact time (from hurdle landing to takeoff) = ",
round(difftime(
as.POSIXct(paste(dt,df$t[posHorizontal],sep=" ")),
as.POSIXct(paste(dt,df$t[posLastMin],sep=" "))),digits = 4)," secs \n",
"Maximum downward flexion of the board = ",
round(max(abs(d$theta2Cum[posLastMin:nextZero])),digits=4)," deg.",
" secs", sep="" ))
Getting this sensor data is absolutely essential for diving coaches and athletes, as it provides them with valuable information about the dive, especially the takeoff component that’s so vital to a successful dive.
Video Capture and Analysis
Video Analysis Overview Currently, diving coaches rely heavily on what they see to give feedback to their divers. They need to be able to associate the data they receive from our sensors with what they see with their own eyes. By correlating sensor data with video, we give coaches the ability to better understand what they’re seeing.
Syncing sensor telemetry with captured video is actually a bit of a challenge. To do so, we turned to a computer vision library, called openCV, to visually determine the exact moment when the diver is at his lowest point, when the board is fully flexed just before takeoff. We then synced that point on the video with the lowest measured point reported by the sensor. The details of this implementation are discussed further in the code sample available on GitHub at bit.ly/2IjjkUO.
Tracking Diver in Flight Computer vision is also used to determine the exact moment when the diver leaves the board. A simple technique called optical flow, often used in object tracking, provides information on the directionality and velocity of a selected point from one image to another. We used the Lucas-Kanade Optical Flow algorithm to determine the time the diver jumps off the board, as well as to determine the speed and the height of the jump. This video-motion analysis (as shown in Figure 7) gives coaches the ability to grasp more insight from each dive.
.png "Tracking Diver Movement in the Air with Video-Motion Analysis")
Figure 7 Tracking Diver Movement in the Air with Video-Motion Analysis
You can get the source code for movement tracking at bit.ly/2QduA7X. Initially, we specify the source video file for our capture:
cap = cv2.VideoCapture("YOUR-VIDEO-FILE.MP4")
Once the file is selected, we prompt the user to select a point in the initial video image to be tracked. With this selected, we call the calcOpticalFlowPyrLK method from OpenCV, which calculates an optical flow for a sparse feature set using the iterative Lucas-Kanade method, as shown in Figure 8.
Figure 8 Calling calcOpticalFlowPyrLK
Code samplecv2.calcOpticalFlowPyrLK(old_gray, gray_frame, old_points,
None, **lk_params)
For visualization, we track the diver on the video:
for i,(new,old) in enumerate(zip(new_points,old_points)):
a,b = new.ravel()
if start is None:
start = a
c,d = old.ravel()
mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)
frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)
if b > lowest_point and abs(a - start) <= THRESH:
low = np.zeros_like(frame)
low = cv2.circle(low, (a,b), 5, color[i+1].tolist(), -1)
lowest_point = b
img = cv2.add(frame,mask) # essey
img = cv2.add(img, low)
cv2.imshow('Frame',img)
old_points = new_points.reshape(-1,2)
Wrapping Up
The goal is for this solution to be easy to use for coaches and invisible to divers. These attributes will allow integration into routine training sessions and make it a standard coaching tool, much the way video playback has been over the past two decades. Coaches will be able to assess the quality of the diver’s takeoff by analyzing key metrics:
- Time in the air during the hurdle indicates how much potential energy is available to the dive.
- Amount of board flexion indicates how effectively the diver has “loaded the springboard” to launch them into the air.
- Time on the board indicates how well the diver rides the flexed springboard and takes advantage of its energy, converting it into height and spin at takeoff.
The benefits of our combined sensor and video approach are clear. From sensors, we gather immediate insight into important takeoff characteristics, allowing coaches to correct technique. Tracking performance indicators over time also allows assessment of changes in diver takeoffs historically and tracks the progress. This feedback loop can act as a powerful incentive for divers to improve their takeoff metrics at each practice, yielding more height and, thus, more time to complete each dive.
The ultimate goal of the project is to provide more detailed knowledge about takeoff and dive biomechanics for our elite U.S. divers, so they can achieve higher scores in competition. To that end, we hope to improve vision analytics, which entails the next steps:
- By using visual distance 1m and 2m markings on the springboard (see Figure 5), we can also estimate the size of the board relative to the trajectory to calculate height of the diver in air.
- Using machine vision and ML to determine exactly when the diver lands on the board and takes off from the board will allow accurate calculation of hurdle time, board-contact time and takeoff instant.
- Consider algorithms beyond Lucas-Kanade, which has some constraints. Lucas-Kanade only works on corners, it doesn’t work well with lighting changes, and it struggles to register large movements.
- Explore implementing body position and pose estimation from captured video.
- Synchronize video with other useful features in the app, so we can trigger actions in the app based on the analyzed video.
Kevin Ashley is a senior architect at Microsoft, and an expert on IoT, machine learning and sports.
Phil Cheetham is currently the senior sport technologist and biomechanist for the U.S. Olympic Committee (USOC) at the Olympic Training Center in Chula Vista, Calif.
Dan Laak was named high performance director for USA Diving in 2018 after serving as head diving coach at the University of Georgia the past 31 years.
Olga Vigdorovich is a data scientist and an avid skier. She works with IoT sensor data analysis and builds data models for scalable cloud platforms based on Microsoft Azure.
Daria Fradkin is a junior at the University of Pennsylvania in the school of Engineering, studying Digital Media Design, a major that combines Computer Science with higher-level courses in Computer Graphics.
Kevin Kang is a fourth-year undergraduate student studying Computer Science with a Data Science option at the University of Washington, Seattle.