Share via

Testing Applications with AppVerifier

  • Article

 

Todd Phillips
Microsoft Corporation

Created: October 2001
Revised: June 2002

Applies to:
     Microsoft® Windows® XP

Summary: AppVerifier is a useful tool for testing applications for compatibility issues with Microsoft Windows XP. This tool can be used to test for a wide variety of known compatibility issues while the application is being exercised. This paper details the steps to using AppVerifier as an effective addition to the application development and testing cycles. (13 printed pages)

Contents

Introduction
Installing the Application Verifier
Testing an Application with AppVerifier
Description of Application Verifier Tests
Reviewing AppVerifier Logs
Conclusion
Related Links

Introduction

Testing applications for compatibility with Microsoft® Windows® XP can be a very complicated process. Fortunately the Application Verifier (AppVerifier) tool developed by the Windows Application and Customer Experience team at Microsoft will assist with this task. AppVerifier encompasses several tools that are specifically designed to test for commonly encountered application compatibility issues as well as some very subtle kernel issues. These tests can also reveal compatibility issues related to the requirements found in the Designed for Windows XP Application Specification.

With the importance of testing your own applications before applying for the Designed for Windows XP logo, many developers will be looking for any tools or guidelines that can ease them through the process. AppVerifier is a valuable tool for working through the testing on common compatibility issues. The tool provides several features that assist in creating more comprehensive application testing a reality.

AppVerifier is a useful tool for identifying some of the common security issues that applications may create in the Windows operating system, such as writing information to incorrect locations within the registry, or the file system, where it could later be modified by a malicious program. Using the Application Verifier will not prevent every possible security or compatibility issue, but it does provide an easy opportunity to avoid and correct the most commonly identified problems.

Installing the Application Verifier

Application Verifier is available from two primary venues: as part of the Application Compatibility Toolkit (ACT), or as a separate download from the Web. Installing the separate tool download is a simple matter of saving the file to a folder on your local hard disk and then running the executable. This will open the setup wizard and install the AppVerifier on your computer.

The installation of AppVerifier is a simple process. One of the basic requirements for using AppVerifier, however, is that you must be installing it on Windows XP. AppVerifier won't install on Microsoft® Windows® 2000.The main window of AppVerifier is shown in Figure 1.

Figure 1. The main working window of AppVerifier

System Requirements for AppVerifier

The system requirements for running AppVerifier are the same as those for running Windows XP:

  • Pentium 233 megahertz (MHz) or equivalent or greater
  • 64 megabytes (MB) RAM (128 MB of RAM recommended)
  • 650 MB of free disk space (more for network installs)
  • VGA monitor
  • Mouse or other supported pointing device
  • CD-ROM or DVD drive

For more information regarding supported hardware, please consult the Hardware Compatibility List (HCL).

Installing AppVerifier

The Application Verifier tool is shipped with Windows XP and can be found in the /Support/Tools folder on the CD-ROM. AppVerifier is also shipped as part of the Application Compatibility Toolkit version 2.0 and higher. Please visit the Windows Application and Customer Experience Web site for the latest version of the Application Compatibility Toolkit or of the Application Verifier. To install AppVerifier from the version of the ACT available with the retail release of Windows XP on your Windows XP system, please use the following steps:

  1. Insert the Windows XP CD-ROM in a local CD-ROM drive.
  2. Use My Computer or Windows Explorer to browse to the CD-ROM drive where you placed Windows XP and open the /Support/Tools folder.
  3. Double-click ACT.EXE to begin the setup program as shown in Figure 2. Accept the defaults for the Setup.

Once the Application Compatibility Toolkit has been installed it can be accessed through the Start menu. AppVerifier can be found in the Application Compatibility Toolkit group on the Start menu.

Figure 2. AppVerifier is installed as part of the Application Compatibility Toolkit.

Testing an Application with AppVerifier

Perhaps the first thing to note about AppVerifier is that it is not an automated test program for your applications. AppVerifier will attach to a program and perform its tests whenever you run the program. It is possible to use AppVerifier and an automated test procedure simultaneously. AppVerifier attaches a "stub," or small piece of code, to the executable program you are testing so that anytime the program is run, AppVerifier will be engaged.

Using AppVerifier to test an application is a relatively simple process. To test an application with AppVerifier, use the following steps:

  1. Open AppVerifier by clicking Start | Programs | Application Compatibility Toolkit | AppVerifier.

  2. Click the Add button to display the Add Application dialog box shown in the following graphic.

    Figure 3. The Add Application dialog box

  3. Browse to the correct folder location for your application and select the executable file to test. Click the Open button when you have the file selected.

  4. Repeat steps 2 and 3 for any additional executables to be tested.

  5. In the Test Settings pane, select the tests you want to perform from the following list:

    • Detect heap corruptions
      This test performs regular checks of the heap and adds guard pages at the end of each allocation to catch possible heap overruns.
    • Locks usage checking
      This test looks for common errors with locks. The output is displayed in a separate debugger application. Note that this test may cause access violations if an error is found.
    • Detect invalid handle usage
      Checks for common problems with handles. The output is displayed in a separate debugger application. Note that this test may cause access violations if an error is found.
    • Thread stack size checking
      This test disables stack growth. This will cause a stack overflow exception if the initial allocation was too small.
    • LogStartAndStop
      This option simply enters log information when the application starts or stops. This helps to make the logs easier to read when reviewing test data.
    • FilePaths
      This test monitors the application's attempts to obtain file path information to see if the program uses hard-coded paths, or a non-standard method of gathering the information. Note that this test may cause the application to crash if an improper method of determining file paths is used.
    • HighVersionLie
      In the past, many applications have been written to run on a single version of Windows. This test will return a very high version number when the application attempts to determine which version of Windows it is running in.
    • RegistryChecks
      This test monitors the application's use of the system registry for any inappropriate or dangerous calls. Any problems detected will be logged.

  6. Click the Options button to select the options you want to use for the testing, as shown in the following graphic. Click the OK button when you have finished.

    Figure 4. Selecting options for testing

  7. Start the application to be tested by either clicking the Run button in AppVerifier or by starting the program normally through the executable or the Start menu.

  8. Exercise the application. Try to use all of the functionality of the program to generate the best data for the AppVerifier logs. Close the application when finished.

  9. View the test results in the AppVerifier log file by clicking the View Logs button in AppVerifier.

The test settings you specify in AppVerifier for a particular application will remain active anytime you run the program until it is removed from the list of applications in AppVerifier. This helps to run programs repeatedly while working out issues.

The first four tests in AppVerifier look for issues that may be found at the kernel level. Because of this, the best output from these tests can only be acquired with the use of a separate kernel debugger. The kernel tests are designed to generate access violation errors when they encounter an error in the program being tested. The kernel debugger will break in at precisely the point in the application's execution when the error is identified. If you run an application through AppVerifier without a debugger attached and one of the kernel tests finds an error, then the application will appear to crash.

To run the app using a debugger, just set all the options and tests desired in AppVerifier, and then launch the application with a debugger according to the directions for that debugger. For example, to debug "myapp.exe" with NTSD (the Windows XP system debugger), go to a command line and type:

ntsd myapp.exe

Any debugger can be used. The assumption is that the user running the tests is familiar with using a debugger. If you are not comfortable with using a debugger, you can run the tests with the Use the appverifier debugger to get crash logs and log kernel checks option turned on. If the program crashes, exhibits unusual behavior, or logs kernel messages, an experienced developer who can then run the application with a debugger should investigate the problems.

Description of Application Verifier Tests

This section will detail the functionality of the tests within Application Verifier. Please note that the lists may not be all-inclusive as the Application Verifier is an ever-growing tool and each version brings with it new tests and expanded capability. This section describes the various functions within each of the six exposed tests in the Application Verifier:

  • Detect heap corruptions
  • Check Lock usage
  • Detect invalid handle usage
  • Check for adequate stack
  • Check version handling
  • Log potential security issues

Page Heap

Page heap detects almost any heap-related bug that is detectable. It is focused on corruption issues and not on leaks. The page heap tests in Application Verifier are not able to capture information about most heap memory leaks, so a different tool would be useful for that category of heap problem.

One of the great advantages of the page heap test in Application Verifier is that many errors can be detected in the instant they happen. For example, an off by one byte error at the end of a dynamically allocated buffer might cause an instant access violation. However, there are a few error categories that cannot be detected instantly and the error report is delayed until the block is freed.

  • Invalid heap pointer—All Win32 and Windows NT® level heap interfaces take as first parameter a pointer to the heap where the operation should happen. The page heap manager detects an invalid heap pointer at the moment of the call.
  • Invalid heap block pointer—After a block is allocated, it can be used as a parameter for several heap interfaces (most notably free() class of interfaces). The page heap manager immediately detects an invalid heap block pointer. See the debugging techniques section for a way to determine if the invalid address is just garbage or is a few bytes off.
  • Multithreaded unsynchronized access to the heap—Some applications call into a heap from multiple threads; this type of scenario requires setting a flag (by the user) that will trigger acquiring a heap lock. The page heap manager will detect this type of violation the moment two threads attempt to call simultaneously into the heap.
  • Assumptions about reallocation of a block at the same address—Although it is thoroughly documented that a reallocation operation is not guaranteed to return the same address, once in a while applications contain this assumption. This is especially the case when the reallocation reduces the size of the block. The page heap manager always allocates a new block during a reallocation and frees the old block. The free block is protected for R/W access and therefore any access to it will raise an access violation.
  • Double free—This is a bug noticed with high frequency. Heap blocks are freed several times. This is detected immediately by the page heap manager because, on the second free, the block will not have the proper prefix header and cannot be found among the allocated blocks. See debugging techniques section for ways to figure out the stack trace of the first free operation. This error can be a variant of the reallocation problem because, when the application frees what it thinks is the address of the block, that block was already freed as part of the reallocation.
  • Access of block after free—The memory blocks freed are kept for a while by the page heap manager in a pool of protected memory. Any access to such a block will raise an access violation. Based on the "locality" principle, we expect to catch most of the problems if the free protected pool is large enough. If the freed block is still in the protected pool, the bug is caught instantly. If the memory was reused, the chances of catching the real culprit decrease considerably.
  • Access after the end of allocated block—The page heap manager places an inaccessible page right after the allocated block. Any access after the end of the block will raise an access violation. There is one little caveat here. Some applications (Microsoft Internet Explorer, for instance) expect allocations to be 8-byte aligned. This is a feature supported since Windows NT 3.5 heap managers. This means that a request size that is not 8-byte aligned will get an 8-byte aligned address and this leaves a few bytes after the end of the block that are still accessible. If the application will corrupt only those few bytes, the error will be caught only when the block gets freed by checking the block suffix pattern.
  • Access before the start of allocated block—The page heap manager can be instructed through a settable flag to place the inaccessible page at the beginning rather than at the end of the block. Any access before the start of the block will raise an access violation.

Locks

The primary purpose of the Locks test is to ensure that the application uses critical sections properly. This includes the following checks:

  • Checking for an unload of a DLL that contains an active critical section. This leads to resource leaks.
  • Checking for a heap block free that contains an active critical section. This also leads to resource leaks.
  • Ensuring that an application does not attempt to initialize a critical section more than once. This can cause undefined behavior.
  • Checking for corruption of a critical section. The DebugInfo field of the critical section must not point to freed memory.
  • Checking for an owner thread ID which is invalid in the current context.
  • Ensuring that the recursion count field of the critical section is not invalid in the current context.
  • Checking to see that the critical section was initialized before being deleted.
  • Ensuring that the critical section is not released more times than the current thread acquired it.
  • Checking to see if the critical section has been initialized prior to being used. Also checking to see that there was no attempt to use the critical section after it had been deleted.
  • Ensuring that the current thread does not attempt to reinitialize the critical section.

It is essential that a debugger be attached to the process in which Locks is running.

Handles

The Handles tests ensure that an application does not attempt to use invalid handles. This includes the following checks:

  • Ensuring that handles are valid when passed to APIs that take a handle. If a NULL value or an INVALID_HANDLE_VALUE value is passed, the handle is clearly invalid and further investigation is necessary.
  • Ensuring that TLS indexes passed to TLS functions are valid.

It is essential that a debugger be attached to the process in which Handles is running.

Stacks

The Stacks tests ensure that an application has sufficient stack space as this test disables stack growth after the initial commit. If an application does not have sufficient stack space, the application will crash with 0xC00000FD (STATUS_STACK_OVERFLOW) and an Application Verifier stop message will not be displayed. This check is critical for services and other system processes which run after the operating system has booted—examples include winlogon, csrss, smss, and services in general.

It is essential that a debugger be attached to the process in which the Stacks test is running.

Remote Procedure Call Checks

The primary purpose of the Remote Procedure Call (RPC) Checks test is to ensure that applications work securely while using RPC. It performs the following checks:

  • Ensuring that an insecure interface is not remotely accessible.
  • Ensuring that a server is not forced to listen on all interfaces, which circumvents a firewall.
  • Ensuring that a server that is remotely accessible does not register an insecure interface.
  • Ensuring that a client does not call a remote endpoint without specifying RPC_C_AUTHN_LEVEL_PKT_PRIVACY.
  • Ensuring that a client does not call a remote endpoint without mutual authentication.

Note These errors do not break into the debugger. You will see an Application Verifier stop message, but DbgBreakPoint does not get called.

It is essential that a debugger be attached to the process in which the RPC Checks test is running. If this is not the case, you will not be able to see the information generated by the test.

HighVersionLie

The HighVersionLie test's primary purpose is to ensure that applications that run on Windows XP will run correctly as future versions of Windows are released. This test is NOT intended for internal system components or applications that ship with the OS. Running this test on the aforementioned components will cause unforeseen results as these components expect a certain version to be returned. This test is designed for applications that will perform version checking and will respond based on the operating system version information returned.

The test is configurable to enable you to provide the major version, minor version, and build number that should be returned to the application.

This test hooks all calls to GetVersion and GetVersionEx.

A debugger is not required.

Security Checks

The primary purpose of the Security Checks test is to ensure that applications work securely in the Windows environment. Fundamentally, this test is looking for various known activities the applications may perform that can open holes in the system security by circumventing accepted security usage. This includes the following:

  • Ensuring that the application does not call the specified APIs below with:

    • NULL DACL

    • DACL which has WRITE_OWNER or WRITE_DAC set with WORLD access

      APIs affected by this function include:

      CreateFileA
      CreateFileW
      CreateDesktopA
      CreateDesktopW
      CreateWindowStationA
      CreateWindowStationW

      RegCreateKeyExA
      RegCreateKeyExW
      RegSaveKeyA
      RegSaveKeyW
      RegSaveKeyExA
      RegSaveKeyExW

      CreateFileMappingA
      CreateFileMappingW
      CreateJobObjectA
      CreateJobObjectW
      CreateThread
      CreateRemoteThread

      CreateDirectoryA
      CreateDirectoryW
      CreateDirectoryExA
      CreateDirectoryExW
      CreateHardLinkA
      CreateHardLinkW
      CreateMailslotA
      CreateMailslotW
      CreateNamedPipeA
      CreateNamedPipeW
      CreatePipe
      CreateMutexA
      CreateMutexW
      CreateSemaphoreA
      CreateSemaphoreW
      CreateWaitableTimerA
      CreateWaitableTimerW

  • Ensuring that the application does not call an API specified below with any of the following conditions:

    • For CreateProcess, lpApplicationName cannot be NULL.

    • For CreateProcess, lpCommandLine cannot contain unquoted spaces. For example, this is invalid:
      c:\program files\foo.exe -t -g c:\program files\foo\bar

    • For WinExec, the command-line cannot contain unquoted spaces. See the example above.

      This affects the following API calls:

      CreateProcessA
      CreateProcessW
      CreateProcessAsUserA
      CreateProcessAsUserW
      WinExec

Reviewing AppVerifier Logs

The whole intent of using AppVerifier is to gather information about possible issues with your application. This information is collected in the AppVerifier log file. To access this information in AppVerifier, click the View Logs button to open the dialog box shown in Figure 5.

Figure 5. The Logs dialog box gives you complete access to the information gathered by AppVerifier.

The Log Results pane displays a tree hierarchy of events logged during the testing of your application. The events can be expanded to reveal more detail. When you highlight a detail in the log, AppVerifier will provide a suggested course of action, or provide some information on the event.

AppVerifier log files can be exported and stored for long-term review by highlighting the log you want to export and clicking the Export Log button. This will open a dialog box that will enable you to browse for a location to save the file. Exporting AppVerifier logs will make it possible for more than one person to review the data generated by testing. The log files are stored as simple text files when exported and will contain a great deal more information than is displayed within AppVerifier's interface.

Conclusion

The Application Verifier (AppVerifier) tool is a useful addition to software testing procedures for Windows-based applications. AppVerifier contains several tests that enable a tester to determine a variety of common application compatibility issues including file path access, registry use, and memory allocation. AppVerifier will assist with several of the common issues that might be encountered when testing applications for "Designed for Windows" status.

AppVerifier is installed as part of the Application Compatibility Toolkit version 2.0, which can be downloaded from Microsoft at the Windows Application and Customer Experience Web page.

For the latest information on Microsoft Windows XP, check out the Windows XP home page.

For more information on the application compatibility technologies in Windows XP, please see the Application Compatibility Web site in the MSDN Library.

For information on application compatibility in Windows for developers and IT professionals, please visit the Windows Application and Customer Experience Web page.

For more information on the "Designed for Windows" program, please visit the Designed for Windows Logo Program Web page.

 

This is a preliminary document and may be changed substantially prior to final commercial release of the software described herein. The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This white paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Acknowledgements

Bill Shihara, Program Manager, Microsoft Corporation

Mark Carroll, Lead Program Manager, Microsoft Corporation