Application Verifier on Drivers; Windows Mobile 6

I tried the various instruction on MSDN and other locations for running Application Verifier on drivers, particularly in cases where no KITL connection is used (such as when running Hopper) and I ran into problems. Documenation corrections have been requested, but I thought I would outline in painful detail how I went about it.

Assumptions

It is assumed that the reader has a bootable device with a KITL connection to Platform Builder (PB).  These instructions will not attempt to define the methods and tools necessary to accomplish this. While these instuctions will eventually lead to the ability to run App Verifier without KITL, you will see that a KITL connection is needed as you develop your solution and configure App Verifier.

I also will not provide an exhaustive discussion regarding the Application Verifier tool; rather I will focuses on the application of this tool to drivers. I will however provide some details you may not have seen before regarding how the various components of Application Verifier interact.

These instructions are applicable in their entirety at the time of this writing, covering both Windows Mobile 5 and Windows Mobile 6; however, issues detected during the development of this document will result in requests for updates that may cause some of the steps outlined to become unnecessary. In other words, watch for documentation updates as new releases of Windows Mobile occur.

The Components Involved in Running Application Verifier

Platform Builder

Platform Builder (PB) will be critical in the early stages of setting up Application Verifier for drivers. While instructions will eventually lead to standalone testing and logging, it is important to take the first steps of testing with a KITL connection in order to avoid frustrations caused by minor setup problems. Feedback in the PB debug window will prove invaluable as you bring up Application Verifier to test a driver.

Application Verifier Remote UI

AppVerifCE.exe can be found at <pb install dir>\cepb\wcetk\ddtk\desktop\AppVerifCe.exe. This tool allows run time setup of application verifier. For drivers it is recommended that auto shimming be setup such that shims are applied at boot time before the drivers are loaded. Although AppVerifCE is not used in this process, it remains useful in reporting the shims applied to a running device for verification purposes.

shim_*.dll

Test DLLs used to perform run-time testing of specified modules; shims are inserted before calls to the test module they shim. They collect information, accept commands, and generate logs. Shims must be loaded prior to the loading of the object they are shimming; this is somewhat problematic for drivers which are loaded early in the boot process, therefore an “autoshimming” process has been defined that allows pre-boot identification of the objects to be shimmed. By default, all available shims will be loaded, but it is generally preferable to specify the specific shims to be loaded based on need. The following is a list of shim_*.dll provided with Windows Mobile:

shim_verifier.dll	Base shim that makes sure function pointers from LoadLibrary/GetProcAddress go to the shims specified; also makes sure you free every dll that you load with LoadLibrary. Shim_verifier will be automatically applied to every module you shim and therefore does not need to be specified in verify.txt.
shim_heap.dll	From MSDN: “Designed to find memory leaks and heap corruption. This shim can expose bugs such as the use of a memory location after freeing the memory location, the freeing of the same memory location twice, buffer overflow, buffer underflow, and bugs associated with moveable memory blocks. This shim can also inject faults into the application.”
shim_hleak.dll	From MSDN: “Designed to find leaks of handle types such as registry, file, event, critical section, mutex, and semaphore handle types.”
shim_usergdi.dll	From MSDN: “Designed to find leaked Graphics Device Interface (GDI) objects and user-defined objects.”
shim_wininet.dll	Used for Internal Microsoft simulation of wininet errors; do not specify this shim
shim_cellcore.dll	Used for Internal Microsoft simulation of cellcore errors; do not specify this shim

Verify.txt

This is the file used to specify the objects to be shimmed and the shimming options. Verify.txt is built as part of the makeimg process when the appropriate environment variables are set, but can be modified with a simple text editor to change test conditions. Registry changes are necessary to insure application of the modifed test conditions; details provided below.

lmemdebug.dll and lmemdebug_autoload.dll

When autoloading shims during the bootload process it is lmemdebug that reads verify.txt, and passes the commands from verify.txt to the shimming process.

lmemdebug.dll is provided in the SAK at

<sak install dir>\public\ossvcs\oak\drivers\lmemdebug

Source code is provided and the binary is automatically built as part of the Windows Mobile build process, however this version has limited application to drivers. Changes to the file are not necessary even if additional shims are added provided the standard shim naming convention is maintained. Note that lmemdebug.dll for Windows mobile is unique from its identically named counterpart found in the Platform Builder binaries. Lmemdebug_autoshim.dll can be found in the Platform builder binaries at <Platform Builder install dir>\CEPB\wcetk\DDTK\<target CPU.\. General embedded help files indicate a need to replace lmemdebug.dll with lmemdebug_autoshim.dll after renaming lmemdebug_autoshim.dll to lmemdebug.dll. This is not necessary in Windows Mobile platforms when accessing files from the Flat Release Directory (FRD) via KITL; however, this is required for standalone testing or when testing with KITL without access to the FRD, therefore it is advisable to simply use lmemdebug_autoshim.dll at all times when debugging drivers. When replacing lmemdebug.dll with lmemdebug_autoshim.dll it is necessary to sign the DLL with a privileged certificate. Other shimming files are signed by premakeimg.bat. Signing can be done manually, or by editing premakeimg.bat. Look for the following line in premakeimg.bat:

      call sign %_FLATRELEASEDIR%\symhlp.dll

Add the following line after this:

   call sign %_FLATRELEASEDIR%\lmemdebug.dll

Shimexp.exe

This handy tool provides a command interface into the shims; this is necessary in the absence of KITL and/or the target command interface provided by shell.exe. Shims log their findings when the process they are shimming exits. Because Drivers never exit it is necessary to send commands to the shims to cause them to log their results; in the absence of a command window, shimexp provides the means for sending these commands. Specific instructions for the use of shimexp follow.

Relfsd.dll

The Release Directory File System Driver (relfsd) extends the release directory to the device as \release when a KITL connection is present. RELFSD impacts autoshimming of drivers in a few ways that will be explained later. Knowledge and control of whether this DLL is included in a build will be crucial. In standard Windows Mobile build tools it is controlled by the environment variable CE_MODULES_RELFSD, however instructions that follow will rely on editing of .bib files for control of whether relfsd is included in the image.

shimeng.dll, htracker.dll, vlog.dll, symhlp.dll

These files contain various functions used by the shimming and logging processes and must be available with the shim DLLs in the FRD when using KITL or on the device image when testing standalone.

Environment Variables

· VERIFY_MODULES is used to specify the objects to be autoshimmed. For instance, in your build window you would enter the command {set VERIFY_MODULES=camera.dll} to test a camera driver contained in camera.dll

· VERIFY_OPTIONS is used to specify options to the shims. Common options used are NoMapFiles EnableFanOut and IgnoreNullFree

· VERIFY_SHIMS is used to specify the shims to be used. If blank, all available shims will be loaded. This environment variable controls which shims are copied to the FRD. When using KITL and FRD access, all shims found in the FRD are loaded. Note that the makeimg process as defined in a standard SAK checks the FRD for the presence of appverif.exe to determine if shimming binaries need to be copied to the FRD. If you change VERIFY_SHIMS and run makeimg, you must delete appverif.exe from the FRD first or the newly specified shims will not be copied. If you want to simply remove shims, you can delete them from the FRD and reboot when using KITL to extract shims from the FRD. Makeimg does not delete shims, so if you change VERIFY_SHIMS to exclude shims used in a previous build, you need to manually delete those shims from the FRD before doing a makeimg. If you are putting shims on the image, you need to insure that this environment variable is set before running makeimg unless the shim DLLs are already in the FRD or the shimming DLLs will be unavailable during the build process. Put simply; delete appverif.exe and shim_*.dll from the FRD and set VERIFY_SHIMS to the desired value before running makeimg.

· IMGSHIMENABLE adds the shim engine to the image; this environment variable should be set to 1 prior to the makeimg process.

Avlogview.exe

This tool can be found in the PB application directory at:

<PB Install dir>\CEPB\wcetk\DDTK\DESKTOP\

Avlogview converts the cryptic logs created by the shims into a more readable format.

Shimming Registry Values

HKLM\ShimAutoload\HasExecuted

This is a breadcrumb set by lmemdebug to indicate whether or not verify.txt has already been processed. When autoshimming takes place, registry values are set corresponding to the shimming options specified in verify.txt; as additional processes are loaded, the autoloader is also loaded, but is able to avoid re-processing of verify.txt. To shim a different driver, without rebuilding and reflashing, it is possible to simply modify verify.txt and reboot, but this registry value must also be set to 0 in order to force a re-read of verify.txt during the reboot. As will be seen below, the method of modifying verify.txt depends on the method of testing being employed.

HKLM\ShimEngine

Contains the shimming settings for each object for which shimming was enabled on a clean boot.

Build Process

getav.bat

In a standard Windows Mobile SAK makeimg.exe calls premakeimg.bat which calls getav.bat. Getav.bat is not distributed in Windows Mobile 5 SAKs, though it is called in the sample premakeimg.bat file. A new batch file must be developed that matches your environment. At the time of this writing there is also an issue in the SAK production process that excludes the application verifier bits from the SAK. They can be found in the General Embedded Windows CE bits if available, or they can be taken from the PB install dir. Because PB is more generally available, a sample getav.bat is provided below that pulls the desired bits from the PB install directory.

@if not defined _ECHOON echo off

REM Gets the App Verifier binaries

setlocal ENABLEDELAYEDEXPANSION ENABLEEXTENSIONS

if not defined VERIFY_SHIMS set VERIFY_SHIMS=shim_heap shim_hleak shim_usergdi

set AVSOURCEDIR=”C:\Program Files\Platform Builder for Windows Mobile\5.00\CEPB\wcetk\DDTK”

set DESTDIR=%_FLATRELEASEDIR%

:copybins

if not exist %DESTDIR%\appverif.exe (

   echo.

   echo Copying AppVerifier binaries...

   xcopy /dy %AVSOURCEDIR%\%_TGTCPU%\appverif*.* %DESTDIR%\

   xcopy /dy %AVSOURCEDIR%\%_TGTCPU%\shimexp.* %DESTDIR%\

   xcopy /dy %AVSOURCEDIR%\%_TGTCPU%\htracker.* %DESTDIR%\

   xcopy /dy %AVSOURCEDIR%\%_TGTCPU%\vlog.* %DESTDIR%\

   xcopy /dy %AVSOURCEDIR%\%_TGTCPU%\symhlp.* %DESTDIR%\

   xcopy /dy %AVSOURCEDIR%\%_TGTCPU%\shim_verifier.* %DESTDIR%\

   for %%i in (%VERIFY_SHIMS%) do xcopy /dy %AVSOURCEDIR%\%_TGTCPU%\%%~ni.* %DESTDIR%\

   if not exist %DESTDIR%\appverif.exe (

      echo Warning: appverifier binaries not correctly copied from %RELEASEPOINT%.

      echo There may have been a MacallanQA build break.

   )

   echo.

)

Endlocal

Note that this simple batch file assumes that PB is installed at

C:\Program Files\Platform Builder for Windows Mobile\5.00\

and may have to be modified accordingly. Note also that this version differs in the following important ways from getav.bat found in the SAK:

1. Shim names in VERIFY_SHIMS can be specified with or without the .dll suffix; the version in the SAK requires that shims be specified without the suffix

2. shim_verify.dll is copied whether specified or not because it is required

3. avsettings.* is not copied; these bits are not required or used in Windows Mobile at this time

The Process of Verifying a Driver

The object is to shim the specified driver(s) with the specified shim(s) and options. Shimming must occur at the time the driver is loaded; relatively early in the boot process. Initial efforts should be employed with KITL enabled and with the FRD available to software running on the device. It may be desirable later to run application verifier on drivers in a standalone device in order to expand the testing to a wider audience and a larger base of scenarios. It is strongly recommended that KITL testing occur first and that an interim step be taken prior to standalone testing to ensure that the device is properly configured because feedback in standalone mode is quite limited.

Three methods of testing will therefore be defined:

1. Testing with KITL and with device access to the FRD

2. Testing with KITL but without access to the FRD; this is used primarily to verify that you are ready for standalone testing

3. Standalone testing without KITL

Testing with KITL and the FRD

At the time of this writing, there is a conflict between test tools lmemdebug.dll and relfsd.dll that will cause a file read error on verify.txt if it is present on the image at the same time a \release directory is found on the device (which occurs with a KITL connection when relfsd.dll is in the image). Therefore, it is important to avoid putting verify.txt in the image at this stage. Actually it will prove very useful to pull verify.txt from the FRD as it allows simple changing of which driver/module is tested without re-building. When utilizing Microsoft build commands and processes, autoshimming of drivers is very simple; just take the following six steps:

1. Do a clean build to create a known good FRD

2. Replace getav.bat with a version similar to what was previously recommended in the getav section

3. Open a build window for this build (i.e. a command prompt that has all needed environment variables set such as can be achieved using the tool bldwnd.exe). Set the following environment variables based on your needs:

a. IMGSHIMENABLE=1

b. VERIFY_MODULES The driver DLLs to be tested

c. VERIFY_OPTIONS There are various shimming options defined in Windows Mobile Documentation; recommended setting:

                                                               i. VERIFY_OPTIONS=NoMapFiles EnableFanout IgnoreNullFree

d. VERIFY_SHIMS The specific shims to be applied; default is {all}

4. Execute a makeimg process; the resultant FRD will contain the necessary image for testing, the specified shim DLLs, and a verify.txt file with the specified shimming options.

5. Based on your development environment either boot from the FRD, or flash your device with the created binary. Boot with KITL enabled and a PB connection. Note that if you are booting from a flashed image on the device, you should go to TargetàConnectivity Options in PB then select “Core Service Settings” and specify “Never (jump to image only)”; also insure that “Enable access to desktop files” is selected in this same configuration window. Make this a clean boot; i.e. one that reformats the tfat partition. This is important because shimming options are stored in the registry. You should see something like the following in your PB debug window:

  25667 PID:e7aa38c6 TID:c79099fa CertVerify: LMemDebug.DLL trust = 2

  26568 PID:e7aa38c6 TID:479099d6 CertVerify: shell.exe trust = 2

  27382 PID:c7909d16 TID:c79099fa APPVERIF -----Shim AutoLoader DllMain '1'

  27383 PID:c7909d16 TID:c79099fa APPVERIF -----++Shim autoloader

  27385 PID:c7909d16 TID:c79099fa APPVERIF -----Setting autoloader breadcrumb...

  27409 PID:c7909d16 TID:c79099fa APPVERIF -----Found shim 'shim_cellcore.dll'

  27443 PID:e7aa38c6 TID:c79099fa RELFSD: Opening file verify.txt from desktop

  27445 PID:c7909d16 TID:c79099fa APPVERIF -----LoadShimForModules: 'shim_cellcore.dll'

6. Shims are designed to log their results when the process they are shimming exits. Because drivers generally don’t exit, it is necessary to send commands to the shims to get them to create logs. Assuming you have loaded shim_heap.dll and are looking for issues reported by this shim, start by establishing a check point, then perform tests that will exercise the driver; finally log a delta from the check point.

a. In the target command window of PB, enter: {loadext shim_heap.dll}. This provides a command interface to the shim

b. Next enter the command: {heapverif * chk}

c. Perform actions on the device to force the driver to be exercised

d. Enter the command: {heapverif * delta}; this will cause a file to be created in the FRD called heapverif_interim.log. Do not attempt to read this file as raw text; rather use the avlogview tool to open it, found at <PB install dir>\CEPB\wcetk\DDTK\DESKTOP. Results will also be dumped to the shell.

The target command window in PB provides the ability to learn more about the shims that are loaded. By simply entering the ? command, shell.exe not only returns help for the command window itself, but for loaded shims as well. For instance, following is the help output for shim_heap.dll:

Shim_heap help:

heapverif <proc>|* [recent] [hist] [chk] [delta] [all] [symbols on|off] [free] [trace on|off] [ignorenull on|off] [breakcnt <count>] [breaksize <size>] [info] [<heap>|*]

   proc process name or pid

   recent list recent allocations

   hist show histogram of allocation sizes

   chk increment checkpoint

   delta report on potential leaks since last checkpoint

   all report on all potential leaks

   symbols turn on/off callstack symbol resolution

   ignorenull ignore freeing a null pointer

   free list blocks in free list

   trace turn on/off debug spew

   breakcnt break into debugger on allocation count

   breaksize break into debugger on allocation size

   info show heap info

   nofinalcheck disable automatic leak checking on process exit

      (useful if you're about to abnormally terminate the process)

 

To change which driver/module is being tested, it is not necessary to do another makeimg after modifying verify.txt. Simply use the remote registry editor from the Tools menu of Platform Builder to set the value of HKLM\ShimAutoload\HasExecuted to 0. Modify verify.txt to specify the modules and/or options desired, then reboot the device with KITL and PB. Verify.txt will be read at boot time and the appropriate commands processed. If you are uncomfortable editing verify.txt directly, you can change the environment variables and run makeimg, but this time consuming exercise is really overkill for what is desired. Various instructions mention running makeverifyfile.bat from the PB install dir; this batch file has the added benefit of replacing lmemdebug.dll with lmemdebug_autoshim.dll which will be necessary in future steps if standalone testing is desired. Makeverifyfile.bat can be found at:

<Platform Builder install dir>\CEPB\wcetk\DDTK\DESKTOP

Note that a version of makeverifyfile.bat is also shipped in the Windows Mobile SAK, but this version does not copy lmemdebug_autoshim.dll. Also, if you use the PB version of Makeverifyfile.bat you will have to manually sign lmemdebug.dll afterward or shimming will fail.

If there are doubts that the shim is loaded into the specified module, run AppVerifCE.exe from <pb install dir>\cepb\wcetk\ddtk\desktop\. When the tool opens, select the Connect button and specify the connection to your device. You will see the currently loaded shims and the modules where they have been applied. In the case of drivers this will likely be device.exe as opposed to the specific DLL containing the driver binaries.

If Your Build Environment differs significantly from Standard SAK

You may find that modifications to your build environment essentially nullify some of the environment variables, the use of makeimg, and/or the getav.bat file. Not to worry; If you have CETK working with the product, initiating Application Verifier on drivers really only involves a few additional requirements.

1. The Application Verifier files must be in the FRD and the EXEs and DLLs must be signed by a privileged certificate; your build process may do this automatically. The standard makeimg process distributed with the SAK will sign all files except lmemdebug.dll

2. Verify.txt must be in the FRD

3. The CETK must be working on your device and Platform Builder must have a connection to your device at boot time

As the device boots, the availability of the FRD as \release on the device will result in a search for lmemdebug.dll and verify.txt; this will then start a process of shimming according to the options specified in verify.txt.

Testing with KITL but without access to FRD

Generally, the purpose for this approach is to prepare for running standalone without KITL. In order to run without access to the FRD, all files must be added to the image. It is recommended that shimming binaries and verify.txt be included in flash in a package that contains files required early in the boot process. Device developers will already have a package for such files generally called OEMXIPKERNEL, though they are free to name it anything they choose. This package will contain nk.exe among other files. Shim files should be added to this package. In Windows Mobile 6, it is necessary to edit two files in the FRD to add the shimming files to the image, 1) the bib file and 2) the package definition file. Typically this will be common.bib and smartfon.cpm.csv. In common.bib, some of the files need to be fixed up to the Kernel space since they are called directly by the Kernel, and others do not. The simplest way to insure that everything goes where it is needed is to put everything in the FILES section of the bib file; the files will then be fixed up to the proper process space when loaded. Since these files are for debug only, this approach, while having a slight negative impact on boot time, is acceptable. Following is an example of files added to common.bib:

FILES

IF IMGSHIMENABLE

   appverif.exe $(_FLATRELEASEDIR)\appverif.exe NK

   lMemDebug.dll $(_FLATRELEASEDIR)\lmemdebug.dll NK

   htracker.dll $(_FLATRELEASEDIR)\htracker.dll NK

   shim_verifier.dll $(_FLATRELEASEDIR)\shim_verifier.dll NK

   symhlp.dll $(_FLATRELEASEDIR)\symhlp.dll NK

   vlog.dll $(_FLATRELEASEDIR)\vlog.dll NK

   shim_heap.dll $(_FLATRELEASEDIR)\shim_heap.dll NK

   shim_hleak.dll $(_FLATRELEASEDIR)\shim_hleak.dll NK

   shim_usergdi.dll $(_FLATRELEASEDIR)\shim_usergdi.dll NK

   verify.txt $(_FLATRELEASEDIR)\verify.txt NK

ENDIF IMGSHIMENABLE

Note the use of the qualifier IMGSHIMENABLE. While not necessary, this makes a convenient way to exclude these files from shipping builds. Note also that shimeng.dll was not called out; this is because the SAK version of common.bib already includes shimeng.dll within an IMGSHIMENABLE qualifier; if you have removed this from your build batch files you will need to add shimeng.dll to the above list.

Corresponding entries must be added to smartfon.cpm.csv:

<BOOTHIVE>appverif.exe,OEMXIPKERNEL

<BOOTHIVE>lMemDebug.dll,OEMXIPKERNEL

<BOOTHIVE>htracker.dll,OEMXIPKERNEL

<BOOTHIVE>shim_verifier.dll,OEMXIPKERNEL

<BOOTHIVE>symhlp.dll,OEMXIPKERNEL

<BOOTHIVE>vlog.dll,OEMXIPKERNEL

<BOOTHIVE>shim_heap.dll,OEMXIPKERNEL

<BOOTHIVE>shim_hleak.dll,OEMXIPKERNEL

<BOOTHIVE>shim_usergdi.dll,OEMXIPKERNEL

<BOOTHIVE>verify.txt,OEMXIPKERNEL

 

As mentioned previously, there is currently a conflict between DLLs that results in an error condition when verify.txt is available both in the FRD and the image. The intent of the currently described approach is to get all files in the image in preparation for standalone testing, but to maintain a KITL connection. Removing verify.txt from the FRD will not resolve the situation because the test in lmemdebug simply looks for the existence of \release on the device and assumes that all files will be found there, therefore if \release is seen on the device and verify.txt is on the image, autoshimming will fail. To avoid this, remove relfsd.dll from the image by removing it from common.bib and smartfon.cpm.csv; this will make the FRD unavailable to the build and force all files to be pulled from the image. In other words, if a KITL connection exists and relfsd.dll is on the image, there will be a directory called \release available and this will cause a file read error of verify.txt. Note that shell.exe, which supports the PB target command window will also be unavailable since it is pulled from the FRD, so there will be no target command window. We could add additional files to the image to address this, but since the ultimate plan is to run standalone, we will employ other methods for sending commands to the shims.

After making the specified changes to the .bib file and .csv file, open a build window, set the environment variables mentioned in the first approach, and perform a makeimg. Flash the image to the device and perform a clean boot with KITL enabled. You should see messages in the debug output window indicating that shims are being loaded.

Shimexp.exe is the tool that provides a command interface to the shims. This file can be included in the image by adding it to the bib file and csv file, but there are other files needed on the device that are likely to be changing during the process of testing, so it is easiest to simply use the remote file viewer to copy the file to \windows on the device. The shims and supporting files could also be put on the device with the remote file viewer (or with ActiveSync explorer), but that would preclude clean boot testing, so it is preferable to put these files in the image. Shimexp does not have a device user interface; it is command line driven, therefore it is necessary to create links to it that provide the required command options. This is easily accomplished with a text editor. Assume that we want to send a chk command to shim_heap.dll. Use notepad to create a text file named “HeapChk.lnk” and put the following text into the file:

77#"\Windows\shimexp.exe" -s shim_heap.dll -f \HeapChk.log -q -c heapverif * chk

Note that the syntax is: [decimal number of characters in the command][#]"\Windows\shimexp.exe" -s [target shim] –f [log file path and name] –q –c [command and options]

A few other useful commands are:

81#"\Windows\shimexp.exe" -s shim_heap.dll -f \HeapDelta.log -q -c heapverif * delta

88#"\Windows\shimexp.exe" -s shim_heap.dll -f \DeviceAll.log -q -c heapverif device.exe all

91#"\Windows\shimexp.exe" -s shim_heap.dll -f \DeviceInfo.log -q -c heapverif device.exe info

95#"\Windows\shimexp.exe" -s shim_heap.dll -f \DeviceRecent.log -q -c heapverif device.exe recent

 

Use the remote file viewer to import these files and put them in:

\Windows\Start Menu\Debug Apps

or a similar location in the Start Menu tree

Select a file from the appropriate folder in the Start Menu to execute the command.

Although verify.txt is on the image, it is still possible to modify it to test other modules or to apply other options without rebuilding. A new verify.txt can be copied to the \windows directory using the remote file viewer and it will shadow the version on the image. As mentioned previously, it will be necessary to set the registry flag HKLM\ShimAutoload\HasExecuted to 0 before rebooting to force the new verify.txt to be read.

Once testing with KITL with all shim files on the image has been successfully accomplished, the same build can be used for standalone testing without KITL. However, this build has another very useful function, it can be passed on to third party driver developers for debugging their drivers. They can establish an ActiveSync connection to the device and setup a connection in PB to use ActiveSync. They can use ActiveSync’s Explore function to copy shimexp.exe and their own version of verify.txt to the device’s \Windows directory and their own shimexp links to the Start Menu. They can use PB’s remote registry editor to clear the HasExecuted registry flag, then reboot.

Testing Standalone

After successfully building an image that contains all of the application verifier files that works with a KITL connection, simply do a clean boot without KITL, copy shimexp.exe and its command links to the device using ActiveSync and start testing. Generic log files will be stored in the root directory of the device and specific log files specified in shimexp links will be stored in the path specified. Log files can be extracted using ActiveSync. Remember to clear HKLM\ShimAutoload\HasExecuted whenever verify.txt is changed. It is not necessary to clear this registry value when rebooting and testing the same modules with the same shims and options.

Warnings and Suggestions

Warnings

Application Verifier does not understand the particular design of a given module. It reports facts before and after particular test conditions. Intimate knowledge of the tested module is necessary to correctly interpret the findings. For instance, a particular driver may load resources the first time it is exercised, and by design not free those resources. Application Verifier might report a “potential leak” because of a delta in resources between the establishment of a check point on the driver and the delta command after the driver is exercised. Additional checkpoints and deltas may be necessary to accurately assess the validity of reported leaks.

Do not attempt to read log files as raw text; the syntax is at times confusing and may lead to erroneous conclusions.

Suggestions

· Application Verifier with KITL is able to integrate well with the overall debug environment, providing useful information on detected exceptions, etc.

· Remove shim_wininet.* and shim_cellcore.* from the FRD; they are not needed and while they do no harm to the shimming process, they do increase boot time and result in unnecessary messages in the debug window.

· To gain the most from this tool it is important to read available documentation found in the Windows Mobile 6 documentation, but beware of documentation intended for the general embedded community of Windows CE; Windows Mobile has unique requirements.

· Use KITL whenever possible; additional debug information and the command interface will speed development.

· When running stand alone, limit testing to a single driver and minimal set of shims in order to reduce impact on performance and resources.

· A single driver may mean multiple DLLs. Consider the case where yourdriver.dll uses a helper dll (driverhelper.dll). You've got some API where you pass a pointer. Yourdriver.dll allocates some memory, that allocation goes through app verifier, but before app verifier passes the call on to the heap manager, it adds some bytes to the request to use as guard bytes to detect buffer overflows. Now, if you pass that pointer to driverhelper.dll, and driverhelper.dll needs to know the size, it will call LocalSize on it. That call will go directly to the heap manager (since driverhelper.dll isn't shimmed), and LocalSize will return the total size (requested by the driver + requested by app veriifer for guard bytes). Driverhelper.dll will write to the entire block (like it should; LocalSize said it's ok), and corrupt the guard bytes. The user gets confused because everything looks good and legal. Short answer -- if you're shimming only specific DLLs, you should also shim any related DLLs. Folks testing applications don't need to worry about this (since everything in the process is shimmed), but you do if you're only testing particular DLLs. EnableFanOut will help in some cases, but not all. As mentioned previously, intimate knowledge of the components being tested is necessary in the effective use of Application Verifier.

· Documentation indicates that development of custom shims is supported, however this is currently not formally documented. Look for support of this capability in the future.

· When booting with KITL and with full access to the FRD, make note of the RELFSD messages in the debug window; this is an indication that a file is being loaded from the FRD. Some of these files, like shell.exe, VMini.dll, and Redir.dll are specific to KITL debugging; others are the files that you will need to load into your image for standalone debugging.

· Use of celog can provide useful debug output in standalone builds, but it is critical to limit the zones logged or the device will bog down and likely not boot. The Debug zone should provide adequate information to assess the boot shimming process. This can be accomplished by adding the following to common.reg:

[HKEY_LOCAL_MACHINE\System\CeLog]

    "Transport"="LocalFile"

    "FileName"="celog.clg"

    "ZoneCE"=dword:800000

It is not recommended to use celog with driver shimming on a regular basis as the impact to performance is quite significant, but this approach can be useful in debugging a failed shimming process.

Example

Assume that the camera driver is called camera.dll and it is suspected of having memory and/or resource leaks. It will be tested after insertion of shim_heap.dll and shim_hleak.dll. The following steps are taken:

1. Clean Build

2. Getav.bat is replaced with a version appropriate to your build environment

3. A build window is opened and the following commands are entered:

a. set IMGSHIMENABLE=1

b. set VERIFY_MODULES=camera.dll

c. set VERIFY_SHIMS=shim_heap.dll shim_hleak.dll

d. set VERIFY_OPTIONS=NoMapFiles EnableFanOut IgnoreNullFree

e. makeimg (your build process may vary; local build experts should be able to translate this approach into instructions specific to your environment)

4. Flash Device with resultant image and establish KITL connection to device with access to the FRD during boot.

5. Shims will autoload and debug output will indicate shimming of camera.dll with output similar to the following:

  28405 PID:a791b666 TID:a791b642 Loading shim 'shim_heap.dll'

  28485 PID:e7aa38c6 TID:a791b642 CertVerify: shim_heap.dll trust = 2

  28571 PID:e7aa38c6 TID:a791b642 CertVerify: symhlp.dll trust = 2

  28641 PID:e7aa38c6 TID:a791b642 CertVerify: vlog.dll trust = 2

  28703 PID:a791b666 TID:a791b642 Applying shim 'shim_heap.dll' to module 'camera.dll'

  28785 PID:a791b666 TID:a791b642 Loading shim 'shim_verifier.dll'

  28857 PID:e7aa38c6 TID:a791b642 CertVerify: shim_verifier.dll trust = 2

  28898 PID:a791b666 TID:a791b642 Applying shim 'shim_verifier.dll' to module 'camera.dll'

  28912 PID:a791b666 TID:a791b642 Enabling shim engine...

  28914 PID:a791b666 TID:a791b642 Verifier loader: SUCCESS

6. Using the command window enter the command: {loadext shim_heap.dll}; this will open a path for the shell to pass commands to the shim. This should result in feedback in the shell similar to the following:

  Windows CE>loadext shim_heap.dll

  DLL=shim_heap.dll, Name=shim_heap.dll

7. Enter the command {heapverif * chk} to establish a baseline for future comparison of resources. You might find the output surprising; rather than a reference to camera.dll, you see a reference to device.exe because the camera driver runs in the device.exe process space. Below is typical output:

Windows CE>heapverif * chk

shim_heap (* chk):

Info for process 'device.exe' 0x07a7abd6, heap 0xffffffff

New heap tracker checkpoint: 1

Note that the “*” tells the shim to iterate through all processes and perform this command on every process that has the shim loaded into it. To limit the command to a single process, use the process name or process id; not the DLL name; in this case camera.dll is loaded into the device.exe process space, so the desired results could be achieved by entering {heapverif device.exe chk}.

8. Utilize the camera in order to exercise the driver.

9. Enter the command {heapverif * delta}; a report follows in the shell similar to the following; in this case no potential leaks were detected; there will also be a heapverif_interim.log in the FRD that can be viewed with avlogview.exe:

Windows CE>heapverif * delta

shim_heap (* delta):

Info for process 'device.exe' 0x07a7abd6, heap 0xffffffff

Process 0x07a7abd6, checkpoint 1

Heap statistics for heap 0x06030000:

   Initial size: 0, Maximum size: 0

   Current bytes: 2604, Max: 8480

   Current count: 7, Max: 13

Checking for leaks in heap 0x06030000...

10. The same process can be repeated for shim_hleak.dll; i.e. {loadext shim_hleak.dll}, {hleak * chk}, exercise driver, {hleak * deltalog} and in fact both shims can be tested simultaneously by loading both shims, establishing checkpoints for each shim, exercising driver, and performing delta on each shim. Note that the hleak shim supports a delta command that dumps to the shell and deltalog that creates a log file whereas the heap shim logs to both shell and log file on the delta command.

11. Introduce various defects into your driver and evaluate the impact on the logs. Try various test conditions such as:

a. Establish check point after starting camera app

b. Take picture and look at delta

c. Take another picture and look at delta (did it increase)

d. Take another picture and look at delta

A driver might reserve resources and not free them for some time; but if there is a steady increase in resources every time you take a picture; you likely have a leak.

Comments

• Anonymous
  April 03, 2007
  Mike Thompson posted a very comprehensive article on this subject at http://blogs.msdn.com/hopperx/archive/2007/03/30/application-verifier-on-drivers-windows-mobile-6.aspx

• Anonymous
  April 15, 2007
  The comment has been removed

• Anonymous
  April 17, 2007
  The comment has been removed

• Anonymous
  May 06, 2007
  Hi, I think this tool is in use during developement stage. But after developement i.e during testing stage. which tools are used for testing the drivers of windows mobile 6.0 completly.

• Anonymous
  May 09, 2007
  Hi Mike, I am pretty sure that the verify.txt is in image and in oemxipkernel package. Just don't know why it can't find it at very early stage. I have another questions, it seems that the shimverifier.dll also checks something other than I specified such as load/unload library. Can I disable it? Thanks, Jack

• Anonymous
  May 10, 2007
  Hi Mike, Another problem is that who loads the lmemdebug.dll? it's too early for it to load because the filesys's API are not ready. So it must be some way to delay of the loading of the lmemdebug to make it successful. Thank you. Jack

• Anonymous
  June 14, 2007
  I am puzzled for one question in here: While doing Hopper test, how can we use appverify to figure out any memory leak in the driver? For example, I set a check point before hopper test as: "heapverif device.exe delta" After hopper stop, I do "heapverif device.exe delta". The list will be long and I an not sure if it make sense for us to analysis it. Appreciate for any comments or suggestions. By the way, is there any way to disable callstack dump while doing delta?

• Anonymous
  June 26, 2007
  Wow, it doesn't pay to let blogs sit; my apologies.   Running app verifier on device.exe is not a good idea.  You effectively run app verifier on every driver and the amount of data will make it difficult to get meaninful results.  Pick "suspect" drivers and focus on them.  You'll notice in my example I shimed camera.dll.  I then performed actions to specifically exercise that driver. And what you want to do is to perform repetitive exercises of the driver; because drivers don't close, the first time they are exercised they may reserve memory that they never free.  App verifier will suspect a leak.  But repeat this process over and over.  If additional memory is consumed each time; you likely have a leak. Mike

• Anonymous
  June 26, 2007
  Jack, I'm afraid I'm at a bit of a loss, but to answer some specific questions with quotes from some experts on the topic:

1. No you can't disable the Load/unload library; if you did, dynamic api redirection wouldn't work (the app would do an end-run around app verifier).
2. lmemdebug.dll is loaded by coredll Is this a platform you are building, or are you trying to apply this to a platform built by someone else?   Mike

• Anonymous
  July 19, 2007
  Hi Mike, Thanks for your article. The first method "Testing with KITL and the FRD" is quite smooth by following your description. But I got a problem when i try the second method- "Testing with KITL but without access to FRD" , all all and txt in common.bib has embedded into my image file beside lmemdebug.dll , surly i has this file in my release folder, and sign it by add commend in premakeimg.bat. I can't not figure it out what's wrong outthere! Any suggestion? Thanks in advance -- Motanis

• Anonymous
  August 01, 2007
  Motanis, sorry for the delayed response.  I've been on a much needed vacation.  I'll need more information.  Are the files added to common.bib actually visable in the windows directory of the booted device?  You should see appverif.exe, lmemdebug.dll, shim_heap.dll, verify.tx and all of the other files added to common.bib.

• Anonymous
  January 28, 2008
  Recently I started using App Verifier to catch memory leaks in our platform and have everything working with KITL and access to the FRD, however, I'm seeing the same thing as a post above, i.e., it complains that verify.txt cannot be found.  I verified that it is indeed in the windows directory, both by using image-explorer and by letting the device boot (which takes a long time since everything seems to be shimmed in the absence of verify.txt).  Any ideas?  Thanks.

• Anonymous
  March 05, 2008
  I have used AppVerifier to verify my device.exe with KITL and FRD several times, and most of them are very successful. However, I found that sometimes they failed because the callstack could NOT be  resolved. The KITL message I saw in the output window of PB5 is as follows: === Couldn't resolve address 0x027ef244 (138521) Couldn't resolve address 0x1e011dd0 (138521) Couldn't resolve address 0x1e01197c (138521) Couldn't resolve address 0x1e011bc8 (138521) === and in the target window after I run "heapverif * delta" is === TrackedItem count: 1 Callstack:   0x07a46b78: shim_heap.dll!(null) + 6b78h   0x07a4984c: shim_heap.dll!(null) + 984ch   0x07a4abbc: shim_heap.dll!(null) + abbch   0x07a4accc: shim_heap.dll!(null) + accch   0x027ef244: UNKNOWN!UNKNOWN + 027ef244h   0x03f55f90: devmgr.dll!(null) + 3f90h   0x03f57298: devmgr.dll!(null) + 5298h   0x03f54254: devmgr.dll!(null) + 2254h   0x03f74c18: coredll.dll!(null) + 16c18h   0x1e011dd0: UNKNOWN!UNKNOWN + 00011dd0h   0x1e01197c: UNKNOWN!UNKNOWN + 0001197ch   0x1e011bc8: UNKNOWN!UNKNOWN + 00011bc8h   0x03f7b368: coredll.dll!(null) + 1d368h Leaked items:   0x00b68800      16 bytes, thread: 0xc712f152, checkpoint 1      Total: 16 bytes === As you see, many "UNKNOWN" shown in the callstack. And then I could NOT know where it is. Sometimes I face this problem, even my friend also face this problem. Do you have any idea why this happen?

• Anonymous
  December 15, 2009
  quote PK: 0x1e011dd0: UNKNOWN!UNKNOWN + 00011dd0h  0x1e01197c: UNKNOWN!UNKNOWN + 0001197ch  0x1e011bc8: UNKNOWN!UNKNOWN + 00011bc8h I came across this probolem too.