Tablet PC

Achieve the Illusion of Handwriting on Paper When Using the Managed INK API

Carlos C. Tapang

Code download available at:TabletPC.exe(186 KB)

This article assumes you're familiar with .NET, the CLR, and C#

Level of Difficulty123

SUMMARY

Creating the illusion of a pen writing on paper is no easy software task. Fortunately, the .NET Framework hosts Tablet PC extensions, which lets you create ink-aware applications for the Tablet PC. This API allows applications to draw strokes on the screen and perform a variety of tasks including document markup, storage, and transmission.

This article shows you how to handle a couple of inking events as used in the InkClipboard sample. Later, it discusses how to avoid common pitfalls including too frequent redrawing, which causes the ink flow to lag behind the pen movements, diminishing the illusion of ink on paper.

Contents

Let Some Ink Flow
The InkCollector Object
A Couple of Inking Events
Slow Lasso in the Ink Clipboard SDK Sample
Ink Lag
Delegating DrawLasso
Conclusion

The Tablet PC is proving to be a fun, exciting platform to write applications for. And creating Tablet-enabled apps is easy thanks to the introduction of a managed SDK that integrates with the Microsoft® .NET Framework. When installed, the SDK provides a number of APIs that encapsulate a lot of the functionality you'll need in a Tablet PC app.

In this article you'll have an opportunity to see how these APIs, specifically the INK API and its managed Ink object, the InkCollector, enhance Tablet PC development. You'll get a chance to use this object in a sample application—namely, the Ink Clipboard program in the Microsoft Tablet PC Platform SDK version 1.1—for a real hands-on experience. You'll also learn to avoid potential pitfalls so that ink flows smoothly and your users experience the simulation of ink on paper.

Let Some Ink Flow

Just as a good understanding of how Win32® messages work helps you write solid apps for Windows®, a good understanding of inking will help you write solid apps for the Tablet PC.

One of the more important things to remember about inking is that the tablet digitizer that measures pen movement is very accurate. Microsoft recommends that OEMs use digitizer grids that have 1,000 lines per inch. Every slight movement of the tip of the pen is detected, even when the tip is not quite touching the digitizer but is hovering over it. As the pen touches the digitizer, data other than grid position may be collected. For example, for pressure-sensitive pens, pressure data is also collected. This means that as long as the pen tip touches the digitizer, even if the pen is not moving, data may be streamed from tablet hardware, collected by the pen driver, and eventually forwarded to applications through the ink objects. This occurs roughly 133 times per second, which is the recommended sampling rate. This data rate is not particularly high compared to current tablet processor speeds, but as you will see later in this article, it has important implications for pen UI design.

The illusion of pen writing on paper is maintained at great computing cost—a cost that doesn't even include recognizing what is written, which is still too expensive to be done on the fly. As the pen tip moves over the digitizer grid, the Tablet PC fires events that exercise both user and system-level code on the input side. On the output side, computations are even more complicated because the screen image has to be accessed for every event. This level of computing would have overwhelmed even the fastest processors a couple of years ago. Luckily for the growing group of Tablet PC aficionados, processors designed for mobile systems can now handle the load. For the application programmer, one tip to remember is to reduce the amount of redrawing or repaints that occur. It is easy to end up repainting more than necessary, which results in "ink lag," an effect that kills the illusion of pen-on-paper. Ink lag occurs when either the ink "flows" while the pen tip is not moving or it trails far behind the tip when the pen is moved fast.

Ink lag is particularly noticeable for tablet orientations other than primary landscape. This is because more computations occur per graphics access in primary portrait, secondary portrait, and secondary landscape modes than in the original, primary landscape mode. The Tablet PC is built on legacy PC graphics hardware, which is designed to work in only one orientation. Other orientations are implemented by applying a transform to the main display buffer at the driver level, increasing the lag (see the sidebar "Tablet PC Display Orientation").

The InkCollector Object

The Tablet PC API makes it possible to write ink-aware apps. This API introduces the concept of ink that allows applications to not only draw strokes on the screen, but also to perform a number of operations on the ink, such as marking up a document, storing it, or transmitting it via the Internet.

The Tablet PC API is designed around pen, ink collection, ink data, and recognition objects. The API is available in both managed and unmanaged forms. The InkCollector object is responsible for collecting ink as the pen writes, as its name implies. Referenced in the InkCollector object is the Ink object, which, in turn, contains Stroke objects.

Ink data is stored in the Ink object, which is referenced in the InkCollector as the Strokes collection property. Each Stroke object in the Strokes collection represents the stream of data collected as the pen moves, from the instant it touches the surface until it is lifted. Figure 1 shows the structure of InkCollector.

Figure 1** InkCollector **

The InkCollector is the base set of pen and gesture input functionality. There are other objects in the INK API. The InkOverlay is a superset of the InkCollector object, and it adds selection and erasing support. The InkPicture is a superset of the InkOverlay object, which adds native support for annotating an image with ink. InkOverlay and InkCollector both need to be attached to a control-based object, while InkPicture is an actual control/component which can be embedded in a form (it has its own GUI). InkPicture is useful for annotating graphic images. InkOverlay is useful for annotating documents whose native data is not ink, for example an Adobe Acrobat document or a PowerPoint® slide.

Sometimes the best way to become familiar with an object such as InkCollector is to dive right in and play with it, so let's walk through some code. If you don't have it already, you can get the Tablet PC Platform SDK version 1.5.

After installing the Tablet PC Platform SDK, in C# under Visual Studio® .NET, add a reference to the "Microsoft Tablet PC API" (Microsoft.Ink.dll), and add a using directive at the beginning of the main C# source file to use the Microsoft.Ink package. Derive your main class from the System.Windows.Forms.Form class, then declare a private InkCollector object in this class:

private InkCollector myInkCollector;

To instantiate an InkCollector, you need a window with which to associate it. Simply use the handle to the form window:

myInkCollector = new InkCollector(this.Handle);

InkCollector has several constructors, and the one I just used accepts a window handle as a parameter.

This InkCollector is initially disabled, so it won't be collecting ink by default. To enable it just set the Enabled property to true:

myInkCollector.Enabled = true;

With just these few statements, you already have an inking application that displays ink as you draw it.

A Couple of Inking Events

Each stroke that's made by the user generates a stroke event. If your application requires that you handle this event (to start the recognizer, for example), you need to write an InkCollectorStrokeEventHandler. To connect your stroke event handler, add the handler delegate to the stroke event as follows:

myInkCollector.Stroke += new InkCollectorStrokeEventHandler(myInkCollector_Stroke);

In the previous statement, InkCollectorStrokeEventHandler is the constructor for the stroke event delegate. Your event handler is myInkCollector_Stroke, which is provided as a parameter to the delegate constructor.

While a stroke is being drawn, a much more frequently occurring event is generated: NewPackets. This event corresponds to a packet or packets of ink data available for the application to process. Think of each packet as simply a data structure whose fields consist of pen coordinates, pressure, and so on. To connect your NewPackets event handler, add a handler delegate to the NewPackets event:

myInkCollector.NewPackets += new InkCollectorNewPacketsEventHandler(myInkCollector_NewPackets);

This statement creates a delegate to the handler you provide, myInkCollector_NewPackets, and adds that delegate to the NewPackets event.

Even if the pen is not moving, as long as it is touching the digitizer, NewPackets events occur at least 133 times per second. Your NewPackets event handler, therefore, has at most only 7.5 milliseconds to process each NewPackets event, which can arrive with one or more packets. At current processor speeds, this is not particularly difficult, but you'll still experience ink lag if your event handler is slow. As it turns out, the Ink Clipboard sample demonstrates how you can run into the ink lag problem.

Figure 2 (which comes from the SDK documentation) shows the inking events that can be triggered by different threads. It is important to know which thread executes which handler so that you can properly determine what that handler can do. For example, the NewPackets handler cannot make System.Windows.Forms.Form method calls because this handler is not executed by the UI thread, but rather by the ink thread. In the table in Figure 2, you can see that the NewPackets event is supposed to fire from the Ink thread, but in the InkClipboard sample it appears as if the NewPackets event is triggered from the UI thread. According to my Tablet PC sources, this depends on whether the main UI thread is declared as single-threaded apartment (STA) or multithreaded apartment (MTA). In STA programs like InkClipboard, the NewPackets event is triggered by the UI thread.

Figure 2 InkCollector Events

Slow Lasso in the Ink Clipboard SDK Sample

The ink clipboard sample uses the InkCollector object to collect ink and allows the user to select ink strokes for copying to the clipboard. To do so, the user draws a selection lasso around the strokes.Tablet PC Display Orientation

Current display chips are designed to stream pixels out of memory in only one manner. At the hardware level, pixels are read from video memory from lower to higher addresses. In raster video, for which current display chips are optimized, the electron beam that draws the pixels starts from the upper-left corner of the monitor. It scans the surface of the monitor from left to right on every row. The rows are drawn from top to bottom. The resulting design naturally assigns bytes in the lowest video memory location to the upper-left pixel. The following bytes are then assigned to pixels in the topmost row, moving from left to right. The following rows of pixels are assigned in the same manner, so that the last, lower-right pixel is assigned to bytes in the highest memory location. There are as many memory allocation schemes as there are color schemes per pixel, but these schemes all boil down to one or more bytes assigned per pixel, with the lowest byte locations assigned to the upper-left pixel, and the highest byte locations assigned to the lower-right pixel.

Display hardware itself, therefore, is limited to one orientation: primary landscape. The other three orientations that the Tablet PC is capable of are all implemented at the display driver level. Bytes in every display buffer or bitmap are arranged in the same manner I just described (from upper-left down to lower-right). The trick is to transform in-memory bitmaps or video buffers before being moved, or sent via DMA, to video memory. This allows for the three additional orientations, but requires that display drivers maintain a transform buffer. Pixels are transform-copied to this buffer from user memory prior to being sent to video display memory. While the driver is doing this transform operation, the graphics I/O system also has to block all user-initiated repaints in order to prevent corruption of the transform buffer. The upshot of all this is that repaints of the screen for orientations other than landscape can be much slower.

The sample implements a lasso by disabling the automatic rendering of ink in the InkCollector, then doing its own rendering when collecting lasso points. (By the way, the InkOverlay object implements a lasso, so an alternative implementation of the concept in this sample would be to use InkOverlay. If I did that, however, I wouldn't have been able to show you how to handle NewPackets and Stroke events.) The sample illustrates the use of handlers quite well for both the Stroke and NewPackets events. It renders the lasso as NewPackets events arrive. For each event, it draws a number of evenly spaced dots, together with a line that changes as the lasso emerges in the form of dots around the selected strokes. The line completes the loop and, for each event, is redrawn from the first dot to the last dot drawn. Figure 3 shows a lasso being drawn around a selection.

Figure 3** Creating a Lasso **

The sample has a slight aesthetic problem. In portrait orientation, the lasso drawing is very slow, causing the lasso dots to lag behind the pen tip. For Tablet PCs capable of secondary portrait and secondary landscape orientations, the problem is evident in the following three orientations: primary portrait, secondary portrait, and secondary landscape. This problem does not occur in primary landscape orientation.

There is no way around the root cause of the problem, which is that display drivers for the Tablet PC have to perform extra buffering for the three new display orientations (see the sidebar "Tablet PC Display Orientation"). Nevertheless, there must be something in the implementation of the lasso that can be improved, for the lag only occurs when a large lasso is drawn quickly. Further examination of the problem reveals four relevant characteristics.

First, it's not only when the pen is moved quickly that the rendering can't keep up. Rendering also slows to a crawl if the pen tip is moved quickly and then suddenly stopped without being lifted off the digitizer. This ink lag is caused by "ink bleed," meaning that the pen tip is collecting ink even when it's not moving.

Second, if the drawing of the connector line is commented out the problem disappears completely, including all three other characteristics. Disabling the drawing of the lasso dots has no effect. Although it would appear as though drawing the connector line is computationally expensive, it's actually a contention problem that slows the NewPackets event handler, as you will see shortly.

Third, if the lasso is drawn fast enough, the lasso dots trail behind the pen tip, and the system fails to collect some ink, as evidenced by the resulting lasso being angular, not smooth (see the bottom part of the lasso in Figure 3). This ink lag occurs because the NewPackets event handler is too slow, as you'll see later.

Finally, the connector line is not being drawn consistently. Sometimes it is not being erased from a previous position, so more than one line appears.

Ink Lag

The myInkCollector_NewPackets routine embodies the NewPackets event handler of the sample. It basically collects stroke points in a collection called lassoPoints, then calls another method, DrawLasso, to draw that part of the lasso, which was collected as a result of a single event. It also draws the line that connects the first point of the lasso stroke to the last. Figure 4 is a code snippet from the myInkCollector_NewPackets event handler that draws the lasso points and the connector line.

Figure 4 Draw Lasso

DrawLassoConnector(0,newPacketIndex-1); for (int i = newPacketIndex; i < lassoPoints.Count; i++) { . . . // some lines elided here dx = ((Point)lassoPoints[i]).X - lastDrawnLassoDot.X; dy = ((Point)lassoPoints[i]).Y - lastDrawnLassoDot.Y; lineLength = Math.Sqrt(dx*dx + dy*dy); // dots are drawn DotSpacing apart if (DotSpacing <= lineLength) { . . . // painting of lasso dot(s) elided } } ••• DrawLassoConnector(0,lassoPoints.Count-1);

DrawLassoConnector, the routine that draws the connecting line, is called twice. The first call erases the previous line, and the second call draws the next incarnation of the line. The first subtle problem with this code is that these two calls occur for every NewPackets event regardless of whether a dot needs to be drawn. The second problem with this code is that the first DrawLassoConnector call does not necessarily remove the previous line drawn by the second call. The second parameter to DrawLassoConnector is an index into the lassoPoints array that determines one of two line endpoints to draw or delete. When deleting the previous line, this index should equal the index for that previous line, and a closer examination of the DrawLasso routine reveals that this is not necessarily the case.

The connector line does not need to be redrawn for every event, and it is being redrawn unnecessarily. Also, as I just mentioned, the second parameter in the first call to DrawLassoConnecter is incorrect. These would explain some of the previously mentioned characteristics of the problem. NewPackets events can occur even when the pen tip is not moving, as long as it is touching the digitizer, so DrawLassoConnector is continually being called with the same start and end points. This, coupled with the fact that the display driver blocks all repaints when it is doing the orientation transformation, exacerbates the resulting ink lag. DrawLassoConnector is trying to draw and redraw the same line about 133 times per second, while the display driver is blocking these redraws almost as often because it is continually calculating transformations. This is all happening within the myInkCollector_NewPackets handler, so the high-priority ink thread that generated the NewPackets event is being blocked longer than necessary. Even though the high-priority ink thread is blocked, the CPU is not overwhelmed because it is able to continue collecting ink at the driver level.

The event handler needs to call DrawLassoConnector only when a new lasso dot is drawn. To fix the ink lag I added a condition to drawing the lasso. I use the first DrawLassoConnector only when a previous line has to be erased, and I also have to ensure that it is indeed erasing a previously drawn line, not drawing a new one.

I introduced a new Boolean private member variable, m_bNewLassoDot. This variable is initialized to false before the lasso stroke is started. (It can be set to false as part of the Stroke event handling.) It is set to true only when a new lasso dot is drawn, so that if this event does not generate new lasso dots, the second call to DrawLassoConnector does not occur and no new lasso line is drawn. It is set to false right after the first call to DrawLassoConnector, thereby ensuring that the first call only occurs if there is a line that needs to be erased. This bool member variable or flag can alternatively be set in the routine DrawLassoDots because you need to draw the connector line only when new lasso dots are drawn. Another private member variable, m_iPreviousEndPointIndex, ensures that the first call to DrawLassoConnector deletes the previously drawn line and does not draw a new line.

The modifications I described take care of the problems relating to the fast-moving pen and the drawing of the connector line. The lasso is able to catch up when the pen tip suddenly stops moving, and no connector lines remain when the lasso completes. However, there are some issues with the third characteristic problem, in which a lasso is drawn quickly and the dots trail behind the pen tip (with the system failing to collect ink). When the pen tip is moved at the right speed, the resulting lasso is still angular, and the lasso dots trail behind the pen tip.

The event handler myInkCollector_NewPackets needs to be faster. To this end, I reduced the number of times DrawLassoConnector is called (see Figure 5). This code prevents the connector line from being drawn if the call to DrawLasso would only produce one lasso dot. Doing so reduces the number of lines drawn per sample, reducing the amount of time the ink thread spends executing DrawLasso. Figure 6 shows the revised code for DrawLasso.

Figure 6 Revised Draw Lasso

/// <summary> /// Draws the selection lasso. /// /// Note that this method does NOT draw a point for every piece /// of packet data. Instead, it draws evenly spaced dots that /// fall along the lasso path. /// </summary> /// <param name="g">The graphics device</param> /// <param name="newPacketIndex">The index where the new packets begin /// </param> public void DrawLasso(int newPacketIndex, int lastNew) { // bail out if lassoPoints array is empty if (lassoPoints.Count == 0) return; // create new client surface descriptor every time Graphics g = CreateGraphics(); // The distance between the current point and the next point int dx, dy; // The distance and angle of the line connecting the current // point and the next point double lineLength; double angle; // The horizontal and vertical spacing between the dots double segSpaceX, segSpaceY; // The total number of dots to draw int totalSegments; // Undraw the line that connected the first point to the // last point before the latest packets were added. if (m_bNewLassoDot) DrawLassoConnector(0, m_iPreviousEndPointIndex); m_bNewLassoDot = false; // For each lasso point, draw the corresponding lasso dots. // Note that this method does NOT draw a point for every piece // of packet data. Instead, it draws evenly spaced dots that // fall along the lasso path. for (int i = newPacketIndex; i < lastNew; i++) { // Always draw the first packet... if (0 == i) { // Update the last drawn lasso dot to this packet lastDrawnLassoDot.X = ((Point)lassoPoints[i]).X; lastDrawnLassoDot.Y = ((Point)lassoPoints[i]).Y; // use a helper method to draw the lasso dot DrawSelectionInkDot(g, lastDrawnLassoDot.X, lastDrawnLassoDot.Y); } else { // Compute the x and y distance between this point and the last // drawn lasso dot dx = ((Point)lassoPoints[i]).X - lastDrawnLassoDot.X; dy = ((Point)lassoPoints[i]).Y - lastDrawnLassoDot.Y; // Compute length of the line between this point and the last // drawn lasso dot lineLength = Math.Sqrt(dx*dx + dy*dy); // If the spacing between this packet and the last drawn lasso // dot is greater than the desired DotSpacing, draw as many // evenly spaced lasso points as needed to fill the distance... if (DotSpacing <= lineLength) { // Got at least a dot m_bNewLassoDot = true; // Calculate the angle of the line connecting this point // to the last drawn lasso point. angle = Math.Atan2(dy, dx); // Calculate the x and y components of the dot spacing segSpaceX = DotSpacing * Math.Cos(angle); segSpaceY = DotSpacing * Math.Sin(angle); // Calculate the number of dots we will need to draw. totalSegments = (int)(lineLength/ (double)DotSpacing); // draw the lasso dots... for (int j = 0; j < totalSegments; j++) { // The coordinates of the lasso dot are determined by the // previous lasso dot plus the x and y components of the // dot spacing (computed above). lastDrawnLassoDot.X = (int)(lastDrawnLassoDot.X + segSpaceX); lastDrawnLassoDot.Y = (int)(lastDrawnLassoDot.Y + segSpaceY); // use a helper method to draw the lasso dot DrawSelectionInkDot(g, lastDrawnLassoDot.X, lastDrawnLassoDot.Y); } } } } // draw the line connecting the first point to the last point if (m_bNewLassoDot) { // Draw the connector line only if there are several dots int currIndex = lastNew - 1; if ((currIndex - m_iPreviousEndPointIndex) > 1) { m_iPreviousEndPointIndex = currIndex; DrawLassoConnector(0, m_iPreviousEndPointIndex); } else { m_bNewLassoDot = false; } } }

Figure 5 Draw the Connector if (m_bNewLassoDot)

{ // Draw the connector line only if there are several dots int currIndex = lastNew - 1; if ((currIndex - m_iPreviousEndPointIndex) > 1) { m_iPreviousEndPointIndex = currIndex; DrawLassoConnector(0, m_iPreviousEndPointIndex); } else { m_bNewLassoDot = false; } }

The code in Figure 6 removes the Graphics parameter and instead declares and creates a Graphics instance within the function. This instance is created whenever the function is called, but it is not explicitly disposed, leaving it to the garbage collector to decide when to dispose this instance. This is not good programming practice because I never reuse any of the instances. To explicitly dispose the Graphics instance, I enclose all lines of code that use the instance in a using construct as follows:

using (Graphics g = CreateGraphics()) { // Put code that uses g here } // g is disposed here

Delegating DrawLasso

To improve the lasso further, I propose moving the routine DrawLasso to the UI thread. (This should help if NewPackets is not already triggered in the UI thread.) In order to accomplish this, I can declare a delegate type, LassoDelegate, and introduce a member variable, m_dAppendLasso, of type LassoDelegate. This will free the event handler of its lasso rendering duty and thereby shorten the execution time of the high-priority ink thread. The myInkCollector_NewPackets routine now simply puts lasso points in the lassoPoints array, then calls the delegate using BeginInvoke. BeginInvoke is a method in the Form control that allows the routine referred to by the delegate (in this case, DrawLasso) to execute in the UI thread. An implementation of this fix is available for download at the link at the top of this article.

Figure 7** Nice Smooth Lasso **

As you saw, the slow lasso problem requires two fixes: getting rid of ink lag and allowing for better concurrency. With the second fix, the lasso is rendered beautifully, regardless of whether the pen tip is moved quickly around the client area (see Figure 7). Even the most discriminating user should be happy with the results.

Conclusion

As you can see, a lot of the ink functionality has been encapsulated for you in the INK API, making development of a simple handwriting collection application a snap. Examining some of the problems encountered when programming with this API gives you a good deal of insight into how the Tablet PC works its magic. With all its benefits and its rising popularity, I'm sure this introduction to Tablet PC programming will come in handy soon.

For related articles see:
Multithreaded Windows Forms Control Sample
Basic Instincts: Programming Events of the Framework Class Libraries

For background information see:
Building Tablet PC Applications by Philip Su and Rob Jarrett (Microsoft Press, 2002)

Carlos C. Tapang was on contract to the Tablet PC team at Microsoft when he wrote this article. He can be reached at ctapang@centerus.com.