January 2012

Volume 27 Number 01

Screen-Based Input - Customized On-Screen Keyboards with the .NET Framework

By Christopher Frenz | January 2012

The need to develop applications that require screen-based inputs has grown steadily in recent years. Developers have traditionally used on-screen data input to let individuals who couldn’t use a computer keyboard enter data or make choices, especially in kiosk applications.

The rapid growth of tablet and mobile computing devices makes on-screen input more pervasive and useful than ever. One need not look past a local coffee shop or mode of public transportation to find people engrossed in the latest “app” for their mobile phone or tablet device. Given the expectations for continued growth in the mobile computing market, it’s not hard to imagine that an understanding of on-screen input methodologies would be of great benefit to any developer.

Moreover, in addition to the mobile market, on-screen keyboards are also getting more popular in applications where security is critical, because screen-based input devices can help to prevent user information from being stolen using hardware-based keyloggers. For example, online stock brokerages such as TradeKing require users to enter all passwords using an on-screen keyboard to help boost security. But even though on-screen keyboards can help to improve security, they do have their own potential risks to consider. While on-screen keyboards mitigate the risk of any hardware-based keyloggers, they’re still potentially susceptible to software-based input logging approaches as well as the much more common technique of “shoulder surfing” to view the input buttons a user presses. For example, one application uses captured video of a person typing into an iPad to identify which keys were typed by aligning the location of the pressed keys with an image of the iPad keyboard (see onforb.es/oobLp2).

Some techniques mitigate the effectiveness of shoulder surfing. One of the most common is a constant randomization of keys to prevent the mapping of captured keypress coordinates to any particular key. In this article, I’ll create an on-screen keyboard that uses key randomization using the Microsoft .NET Framework. The application will allow the on-screen keyboard to be linked to the particular appli­cation for which on-screen input is desirable. However, this article describes the techniques needed to build an on-screen keyboard, and the sample application is designed to illustrate these techniques, not to provide a fully featured keyboard application.

The On-Screen Keyboard GUI

The first step is laying out the GUI, which includes a “key” for each letter of the alphabet and each number (see Figure 1). Before you start dragging controls onto your form, however, there are a few issues to consider. In most .NET applications (and other applications as well), whenever you want the user to submit input by clicking on something, the standard control to use would be a button. But you can’t use a button control in an on-screen keyboard because on being clicked, a button control automatically gains the focus of the Windows OS. Because keyboard input is meant to go to the active foreground application (that is, the application with the focus), an on-screen keyboard should never gain the focus of the OS. Rather than use button controls for keys, I’ll use label controls instead, because they support a Click event as buttons do, but unlike buttons aren’t allowed to gain focus. Labels therefore make an ideal choice, being able to respond to clicks without initiating a change in application focus—with a bit of extra coding that you’ll see in the next section. For purposes of the sample application, these labels will be named Label1 through Label36. A textbox, TextBox1, is also created to compare user inputs entered into the on-screen keyboard with those that appear in the external application (see Figure 1). 

The Design View of the On-Screen Keyboard GUI
Figure 1 The Design View of the On-Screen Keyboard GUI

Staying Out of Focus

Although using a control such as a label is necessary to avoid causing the application to receive the focus of the OS, it’s not enough in itself because the on-screen keyboard application can also receive the focus when the form itself is loaded and when the form or any control on the form is clicked by the mouse. To rectify this situation, I need to add some additional code to the keyboard application. 

First, to prevent the form from gaining the focus when it’s first loaded, I’ll add the following code (I’m using Visual Basic .NET) to the application:

Private Const WS_EX_NOACTIVATE As Integer = &H8000000
Protected Overrides ReadOnly Property CreateParams() As CreateParams
  Get
    CreateParams = MyBase.CreateParams
    CreateParams.ExStyle = CreateParams.ExStyle And WS_EX_NOACTIVATE
    Return CreateParams
  End Get
End Property

This code overrides the form’s CreateParams property, which is used in the creation of the form object. By overriding this property with the WS_EX_NOACTIVATE window style, I prevent the form from coming to the foreground upon loading, which means that loading the on-screen keyboard won’t take the focus away from whatever other application was active at the time of launch. Once this code is put in place, it’s important to next ensure that the application can’t achieve focus via a mouse click. Adding this code accomplishes that:

Private Const WM_MOUSEACTIVATE As Integer = &H21
Private Const MA_NOACTIVATE As Integer = &H3
Protected Overrides Sub WndProc(ByRef m As Message)
  If (m.Msg = WM_MOUSEACTIVATE) Then
      m.Result = MA_NOACTIVATE
    Else
      MyBase.WndProc(m)
    End If
End Sub

This code overrides the form’s WndProc function, which the form uses to receive all user input. The overriding function intercepts WM_MOUSEACTIVATE messages, which are sent when an inactive window is clicked on. It also ensures that the on-screen keyboard application doesn’t gain the focus as a result of the mouse click by setting the function return value to MA_NOACTIVATE. The “Else” condition of this code ensures that all other mouse input messages are passed through, thereby allowing the on-screen keyboard application to detect label clicks without ever gaining focus.

Adding the Keyboard Functionality

At this point, I have a GUI for the application and code to ensure that it remains focus-free. It’s now time to add the actual keyboard functionality. The first bit of this functionality will create a subroutine (AssignKeys) to assign random but unique keys to each label (see Figure 2).

Figure 2 Assigning Random, Unique Keys to Each Label Control

Private Sub AssignKeys()
  Dim Character() As Char =
    {"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", _
     "Q", "W", "E", "R", "T", "Y", "U", "I", "O", "P", _
     "A", "S", "D", "F", "G", "H", "J", "K", "L", "Z", _
     "X", "C", "V", "B", "N", "M"}
  Dim Keys(36) As Char
  Dim I, X As Integer
  Dim Rand As New Random()
  Dim Used(36) As Integer
  Dim Unique As Boolean = False
    Used(0) = -1
    For I = 0 To 35
      Unique = False
      X = Rand.Next(0, 36)
      If Character(X) <> " " Then
        Keys(I) = Character(X)
        Character(X) = " "
      Else
        Do Until Unique = True
          X = Rand.Next(0, 36)
          If Character(X) <> " " Then
            Keys(I) = Character(X)
            Character(X) = " "
            Unique = True
          End If
        Loop
        End If
  Next
    Label1.Text = Keys(0)
    Label2.Text = Keys(1)
    Label3.Text = Keys(2)
    Label4.Text = Keys(3)
    Label5.Text = Keys(4)
    Label6.Text = Keys(5)
    Label7.Text = Keys(6)
    Label8.Text = Keys(7)
    Label9.Text = Keys(8)
    Label10.Text = Keys(9)
    Label11.Text = Keys(10)
    Label12.Text = Keys(11)
    Label13.Text = Keys(12)
    Label14.Text = Keys(13)
    Label15.Text = Keys(14)
    Label16.Text = Keys(15)
    Label17.Text = Keys(16)
    Label18.Text = Keys(17)
    Label19.Text = Keys(18)
    Label20.Text = Keys(19)
    Label21.Text = Keys(20)
    Label22.Text = Keys(21)
    Label23.Text = Keys(22)
    Label24.Text = Keys(23)
    Label25.Text = Keys(24)
    Label26.Text = Keys(25)
    Label27.Text = Keys(26)
    Label28.Text = Keys(27)
    Label29.Text = Keys(28)
    Label30.Text = Keys(29)
    Label31.Text = Keys(30)
    Label32.Text = Keys(31)
    Label33.Text = Keys(32)
    Label34.Text = Keys(33)
    Label35.Text = Keys(34)
    Label36.Text = Keys(35)
  End Sub

The routine in Figure 2 creates an array (Character) that contains all of the alphanumeric characters selected to appear on the keyboard and then applies the random number generator to select a random element from this array. As long as the element hasn't been previously selected, the character stored in that particular element of the array is copied to an array called Keys. This process repeats until all 36 characters are assigned to the Keys array, which randomizes the location of each character in the array. Once the array is randomized, the elements of the Keys array are assigned to the Text property of each Label to allow their assigned characters to be displayed on the screen. This AssignKeys subroutine is initially called on the execution of the Form_Load event.

Now that the characters have been assigned to their respective keys, I need to add code that handles converting the mouse clicks on the on-screen keyboard application into the equivalent of sending a keypress to the target application. To do this, I need to make use of the user32.dll API, which Windows uses to handle many UI-related functions such as window handling and other window management functions. To set the application up to properly make use of this API functionality, I’ll add the following DLL Import statements to the Form class:

<DllImport("user32.dll", SetLastError:=True)> _
Private Shared Function FindWindow(ByVal lpClassName As String, _
    ByVal lpWindowName As String) As IntPtr
End Function
<DllImport("user32.dll", SetLastError:=True)> _
Private Shared Function SetForegroundWindow(ByVal hWnd As IntPtr) As Boolean
End Function

The user32.dll API will be used every time a Label_Click event is launched, because each such event will resemble the code in Figure 3.

Figure 3 Using the user32.dll API for Label_Click Events

Private Sub Label1_Click(ByVal sender As System.Object,_
  ByVal e As System.EventArgs) Handles Label1.Click
  Dim X As Char
  X = CChar(Label1.Text)
  Dim theHandle As IntPtr
  theHandle = FindWindow(Nothing, "Untitled - Notepad")
  If theHandle <> IntPtr.Zero Then
    SetForegroundWindow(theHandle)
    SendKeys.Send(X)
  End If
  TextBox1.Text = TextBox1.Text & Label1.Text
  AssignKeys()
End Sub

When a label click event arises, the variable “theHandle” is used to store the application handle for the application to which the on-screen keyboard is going to send its input. In this case, the application handle was set to a freshly loaded copy of Notepad because it’s universally available on all Windows systems. If the application handle is currently present on the system, the application bearing that handle (Notepad) moves to the foreground and the character assigned to that label is sent to the application. The character is also appended to any text found within the keyboard application’s textbox to demonstrate that the characters that appear in Notepad are the same characters received by the keyboard application itself. As a last step, the AssignKeys subroutine is called again to re-randomize the key positions and make shoulder surfing even more difficult. This procedure is illustrated in Figure 4 and Figure 5, where Figure 4 shows a newly loaded version of the application and Figure 5 demonstrates the on-screen keyboard and Notepad after several keypresses on the on-screen keyboard.

A New Instance of the On-Screen Keyboard
Figure 4 A New Instance of the On-Screen Keyboard

The On-Screen Keyboard Sending Input to Notepad
Figure 5: The On-Screen Keyboard Sending Input to Notepad

Enhanced Security and Mobile Porting

This article demonstrated the development of an on-screen keyboard using the .NET Framework. I hope it provided some insight into how on-screen keyboards can be developed for use in improving the security of certain elements of data entry or for use in porting .NET applications to mobile platforms.


Christopher M. Frenz is the author of the programming books, “Visual Basic and Visual Basic .NET for Scientists and Engineers” (Apress, 2002) and “Pro Perl Parsing” (Apress, 2005). He can be reached at cfrenz@gmail.com.

Thanks to the following technical expert for reviewing this article: Robert Green