October 2010

Volume 25 Number 10

UI Frontiers - Multi-Touch Inertia

By Charles Petzold | October 2010

UIs on computers seem to be most appealing when they mask the digital nature of the underlying technology, and instead mimic the analog feel of the real world. This trend began when graphical interfaces started replacing command lines, and then continued with the increased use of photographs, sound and other media.

The incorporation of multi-touch into video displays has given an extra boost to the process of making user controls more lifelike and intuitive. I think you’ve only begun to glimpse the potential of the touch revolution to revamp your relationship with the computer screen.

Perhaps someday we will even eliminate one of the ugliest vestiges of digital technology known to man: the push-button volume control. The basic dial that controlled volume (and other settings) on radios and televisions was delightfully simple, but has been replaced with awkward push buttons on remote controls and the appliances themselves.

Computer interfaces often offer sliders for volume control. These are almost as good as dials. But push-button volume controls are still persistent. Even the multi-touch screen of the Zune HD wants you to press plus and minus signs to increase and decrease volume. More preferable would be a dial that responds to touch.

I like the dial for multi-touch. Perhaps that’s why I focused on simulating dials in my article, “Finger Style: Exploring Multi-Touch Support in Silverlight,” in the March 2010 issue of MSDN Magazine (msdn.microsoft.com/magazine/ee336026), and why I’m returning to dials to talk about handling multi-touch inertia in the Windows Presentation Foundation (WPF).

The Inertia Event

One of the ways in which a multi-touch interface attempts to mimic the real world is by introducing inertia—the tendency for objects to maintain the same velocity unless acted upon by other forces, such as friction. In multi-touch interfaces, inertia can keep a visual object moving when the fingers have left the screen. The most common application of touch inertia is in navigating lists, and inertia is already built into the WPF ListBox.

Among the downloadable code for this article is a little program called IntertialListBoxDemo that simply fills a WPF ListBox with a bunch of items so you can test it out on your multi-touch display.

In your own WPF controls, you’ll need to decide how much inertia you want, if any. Handling inertia is easiest if you just want inertia to cause the same response from your program as direct manipulation. The hard part is getting a good understanding of the numeric values involved.

As you’ve seen in previous articles, a WPF element is informed of the onset of multi-touch manipulation with two events: ManipulationStarting (which is a good opportunity for performing some initialization) and ManipulationStarted. These two events are then followed by potentially very many ManipulationDelta events, which consolidate the action of one, two or more fingers into translation, scaling and rotation information.

When all the fingers have lifted from the element being manipulated, a ManipulationInertiaStarting event occurs. This is your program’s opportunity to specify how much inertia you want. (I’ll discuss how, shortly.) The effect of inertia takes the form of additional ManipulationDelta events until the object “runs down” and a ManipulationCompleted event signals the end. If you don’t do anything with ManipulationInertiaStarting, it’s followed immediately by ManipulationCompleted.

The use of the ManipulationDelta event to indicate both direct manipulations and inertia makes this whole scheme extremely easy to use. Basically, you can implement inertia with a single statement in the ManipulationInertiaStarting event. If necessary, you can discern the difference between direct manipulation and inertia by the IsInertial property of ManipulationDeltaEventArgs.

As you’ll see, you specify how much inertia you want in one of two different ways, but they both involve deceleration—a decrease in velocity over a period of time until the velocity reaches zero and the object stops moving. This involves some concepts in physics that most programmers don’t encounter often, so perhaps a little refresher course might help.

A Refresher on Acceleration

If something is moving, it is said to have a speed or velocity that can be expressed in units of distance-per-time. If the velocity is itself changing over the course of time, then the object has acceleration. A negative acceleration (indicating decreasing velocity) is often called deceleration.

Throughout this discussion, let’s assume that the acceleration or deceleration itself is constant. A constant acceleration means that velocity changes linearly—an equal amount per unit of time. If acceleration is 0, the velocity remains constant.

For example, automobile manufacturers sometimes advertise their cars as being able to accelerate from 0 to 60 mph in a certain number of seconds, let’s say, 8 seconds. The car is initially at rest but by the end of 8 seconds, it’s going at 60 mph, as shown in Figure 1.

Figure 1 Acceleration

Notice that the velocity increases by the same amount every second. The acceleration is expressed as the change in velocity during a particular period of time, or in this case, 7.5 mph per second.

That acceleration value is a little awkward because there are two units of time involved: hours and seconds. Let’s get rid of hours and just use seconds. Because 60 mph is 88 feet per second, the car could equivalently be said to go from 0 to 88 feet per second in 8 seconds. The acceleration is 11 feet per second per second, or (as it’s often phrased) 11 feet per second squared.

We can go back to miles and hours, but the units get ridiculous as the acceleration is calculated as 27,000 mph squared. This implies that at the end of the hour, the car is traveling 27,000 mph. Of course it doesn’t do that. In real life, as soon as the car gets up to 60 mph or thereabouts, the velocity levels off and the acceleration goes down to 0. That’s a change in acceleration, which in engineering circles is referred to as a jerk.

But for this exercise we’re assuming constant acceleration. Starting at a velocity of 0, as a result of acceleration a, the distance x traveled during time t is:

The ½ and square are necessary because the velocity v is calculated as the first derivative of the distance with respect to time:

If a is 11 feet per second squared, then at tequal to 1 second, the velocity is 11 feet per second, but the car has only traveled 5.5 feet. At t equal to 2 seconds, the velocity is 22 feet per second, and the car has traveled a total of 22 feet. During each second, the car travels a distance based on the average velocity during that second. At tequal to 8 seconds, the velocity is 88 feet per second, and the car has traveled a total of 352 feet.

Multi-touch inertia is like watching the car in reverse: At the time the fingers lift from the screen, the object has a certain velocity. The application specifies a deceleration that causes the velocity of the object to decrease linearly down to 0. The higher the deceleration, the quicker the object stops. A deceleration of 0 causes the object to continue moving at the same velocity forever.

Two Ways to Decelerate

The ManipulationDelta event arguments include a Velocities property of type ManipulationVelocities, which itself has three properties corresponding to the three types of manipulation:

• LinearVelocity of type Vector
• ExpansionVelocity of type Vector
• AngularVelocity of type double

The first two properties are expressed as device-independent units per millisecond; the third is in angles (in degrees) per millisecond. Of course, the millisecond is not an intuitively comprehensible period of time, so if you need to look at these values and get a sense of what they mean, you’ll want to multiply them by 1,000 to convert to seconds.

Applications don’t often make use of these Velocity properties, but they provide initial velocities for inertia. When the user’s fingers leave the screen, a ManipulationInertiaStarting event occurs. The event arguments include these three properties that let you specify inertia separately for those same three types of manipulation:

• TranslationBehavior of type InertiaTranslationBehavior
• ExpansionBehavior of type InertiaExpansionBehavior
• RotationBehavior of type InertiaRotationBehavior

Each of those classes has an InitialVelocity property, a DesiredDeceleration property, and a third property named DesiredDisplacement, DesiredExpansion or DesiredRotation, depending on the class.

For translation and rotation, all the Desired properties have default values of Double.NaN, that special bit configuration that indicates “not a number.” For expansion, the Desired properties are of type Vector with X and Y values of Double.NaN, but the concept is the same.

Let’s focus on rotational inertia first, because it’s an effect familiar from the real world (such as a playground roundabout), and you don’t have to worry about objects flying off the screen.

By the time you get a ManipulationInertiaStarting event, the InertiaRotationBehavior object has been created, and the InitialVelocity value has been set from the previous ManipulationDelta event. If the InitialVelocity is 1.08, for example, that’s 1.08 degrees per millisecond, or 1,080 degrees per second, or three revolutions per second, or 180 rpm.

To keep the object spinning, you set either DesiredRotation or DesiredDeceleration but not both. If you try to set both, the last one you set will be valid and the other will be Double.NaN.

The first option is to set DesiredRotation to a value in degrees. This is the number of degrees the object will spin before it stops. If you set DesiredRotation to 360, for example, the spinning object will make just one additional revolution, slowing down and stopping in the process. The advantage is that you get the same amount of inertial activity regardless of the initial speed, so it’s easy to predict what will happen. The disadvantage is that it’s not entirely natural.

The alternative is to set DesiredDeceleration to a value in units of degrees per millisecond squared, and here’s where it becomes a bit slippery because it’s hard to guess at a good value.

If InitialVelocity is 1.08 degrees per millisecond and you set DesiredDeceleration to 0.01 degrees per millisecond squared, then the velocity will decrease by 0.01 degrees every millisecond: to 1.07 degrees per millisecond at the end of the first millisecond, 1.06 degrees per millisecond at the end of the second millisecond, and so forth linearly until the velocity gets down to 0. That whole process will take 108 ms, or a little more than one-tenth of a second.

You probably want to set DesiredDeceleration to something less than that, perhaps 0.001 degrees per millisecond squared, which will cause the object to continue spinning for 1.08 seconds.

Watch out if you want to convert the deceleration units to something a little easier for the human mind to contemplate. A velocity of 1.08 degrees per millisecond is the same as 1,080 degrees per second, but a deceleration of 0.001 degrees per millisecond squared is 1,000 degrees per second squared. To convert deceleration to seconds you need to multiply by 1,000 twice because time is squared.

If you combine the two formulas shown earlier and eliminate t, you get:

This implies that the two methods for setting deceleration are equivalent and can be converted from one to the other. If the InitialVelocity is 1.08 degrees per millisecond, and you set the DesiredDeceleration to 0.001 degrees per millisecond squared, that’s equivalent to setting DesiredRotation to 583.2 degrees. In either case, rotation stops after 1,080 ms.

Experimentation

To get a feel for rotational inertia, I built the RotationalInertiaDemo program shown in Figure 2.

Figure 2 RotationalInertiaDemo Program in Action

The wheel on the left is what you turn with your finger, either clockwise or counter-clockwise. It’s very simple: just a UserControl derivative with two Ellipse elements in a Grid with all the Manipulation events handled in MainWindow.

The ManipulationStarting event performs initialization by restricting the manipulation to rotation only, allowing single-finger rotation, and setting the center of rotation:

args.IsSingleTouchEnabled = true;
args.Mode = ManipulationModes.Rotate;
args.Pivot = new ManipulationPivot(
             new Point(ctrl.ActualWidth / 2, 
                       ctrl.ActualHeight / 2), 50);

The speedometer at the right is a class called ValueMeter, and shows the current velocity of the wheel. If it looks vaguely familiar, it’s only because it’s an enhanced version of a ProgressBar template from an article I wrote for this magazine more than three years ago. The enhancements involved some flexibility with the labels so I could use it to show velocity in four different units. The GroupBox in the middle of the window let you select those units.

When your finger is rotating the dial, the meter shows the current angular velocity available from the Velocities.AngularVelocity sub-property of the ManipulationDelta event arguments. But I found it wasn’t possible to funnel the velocity of the dial directly to ValueMeter. The result was too jittery. I had to write a little ValueSmoother class to perform a weighted average of all the values from the past quarter second. The ManipulationDelta event handler also sets a RotateTransform object that actually rotates the dial:

rotate.Angle += args.DeltaManipulation.Rotation;

Finally, the Slider at the bottom lets you select a deceleration value. The value of the Slider is only read when the finger leaves the dial and a ManipulationInertiaStarted event is fired:

args.RotationBehavior.DesiredDeceleration = slider.Value;

That’s the entire body of the ManipulationInertiaStarted event handler. During the inertial phase of the manipulation, the velocity values are much more regular and don’t have to be smoothed, so the ManipulationDelta handler uses the IsInertial property to determine when to pass velocity values directly to the meter.

Boundaries and Bounce Back

The most common use of inertia with multi-touch is in moving objects around the screen, such as scrolling through a long list or flicking elements to the side. The big problem is that it’s very easy to cause an element to go flying right off the screen!

But in the process of handling that little problem, you’ll discover that WPF has a built-in feature that makes manipulation inertia even more realistic. You may have noticed earlier in the ListBoxDemo program that when you let inertia scroll to the end or beginning of the list, the entire window bounces a bit when the ListBox reaches the end. That’s also an effect you can get in your own applications (if you want it).

The BoundaryDemo program consists of just one ellipse with an identity MatrixTransform set to its RenderTransform property sitting in a Grid named mainGrid. Only translation is enabled during the OnManipulationStarting override. The OnManipulationInertiaStarting method sets inertia deceleration like so:

args.TranslationBehavior.DesiredDeceleration = 0.0001;

That’s 0.0001 device-independent units per millisecond squared, or 100 device-independent units per second squared, or about 1 inch per second squared.

The OnManipulationDelta override is shown in Figure 3. Notice the special handling when the IsInertial is true. The idea behind the code is that the translation factors should be attenuated when the ellipse has drifted partially off the screen. The attenuation factor is 0 if the ellipse is within mainGrid, and goes up to 1 when the ellipse is halfway beyond the boundary of mainGrid. The attenuation factor is then applied to the translation vector delivered to the method (called totalTranslate) to calculate usableTranslate, which provides the values actually applied to the transform matrix.

Figure 3 The OnManipulationDelta Event in BoundaryDemo

protected override void OnManipulationDelta(
  ManipulationDeltaEventArgs args) {
  FrameworkElement element = 
    args.Source as FrameworkElement;
  MatrixTransform xform = 
    element.RenderTransform as MatrixTransform;
  Matrix matx = xform.Matrix;
  Vector totalTranslate = 
    args.DeltaManipulation.Translation;
  Vector usableTranslate = totalTranslate;
  if (args.IsInertial) {
    double xAttenuation = 0, yAttenuation = 0, attenuation = 0;
    if (matx.OffsetX < 0)
      xAttenuation = -matx.OffsetX;
    else
      xAttenuation = matx.OffsetX + 
        element.ActualWidth - mainGrid.ActualWidth;
    if (matx.OffsetY < 0)
      yAttenuation = -matx.OffsetY;
    else
      yAttenuation = matx.OffsetY + 
        element.ActualHeight - mainGrid.ActualHeight;
    xAttenuation = Math.Max(0, Math.Min(
      1, xAttenuation / (element.ActualWidth / 2)));
    yAttenuation = Math.Max(0, Math.Min(
      1, yAttenuation / (element.ActualHeight / 2)));
    attenuation = Math.Max(xAttenuation, yAttenuation);
    if (attenuation > 0) {
      usableTranslate.X = 
        (1 - attenuation) * totalTranslate.X;
      usableTranslate.Y = 
        (1 - attenuation) * totalTranslate.Y;
      if (totalTranslate != usableTranslate)
        args.ReportBoundaryFeedback(
          new ManipulationDelta(totalTranslate – 
          usableTranslate, 0, new Vector(), new Vector()));
      if (attenuation > 0.99)
        args.Complete();
    }
  }
  matx.Translate(usableTranslate.X, usableTranslate.Y);
  xform.Matrix = matx;
  args.Handled = true;
  base.OnManipulationDelta(args);
}

The resulting effect is that the ellipse doesn’t stop immediately when it hits the boundary, but slows down dramatically as if encountering some thick sludge.

The OnManipulationDelta override also calls the ReportBoundaryFeedback method, passing to it the unused portion of the translation vector, which is the vector totalTranslate minus usableTranslate. By default, this is processed to make the window bounce a bit as the ellipse slows down, demonstrating the physics principle that for every action there’s an opposite and equal reaction.

This effect works best when the velocity on impact is fairly large. If the ellipse has slowed down appreciably, there’s a vibration effect that’s less satisfactory, but it’s something you might want to control in more detail. If you don’t want the effect at all (or only want it in some cases), you can simply avoid calling ReportBoundaryFeedback. Or you can handle the ManipulationBoundaryFeedback event yourself. You can inhibit default processing by setting the Handled property of the event arguments to true, or you can choose another approach. I’ve left an empty OnManipulationBoundaryFeedback method in the program for your experimentation.

Charles Petzold  is a longtime contributing editor to MSDN Magazine*. He’s currently writing “Programming Windows Phone 7” (Microsoft Press), which will be published as a free downloadable e-book in the fall of 2010. A preview edition is currently available through his Web site charlespetzold.com.*

Thanks to the following technical experts for reviewing this article: Doug Kramer and Robert Levy