Using Application Verifier Within Your Software Development Lifecycle

  • 아티클

 

Microsoft Corporation

Revised October 2006

Contents

Introduction
What Is AppVerifier?
AppVerifier Features
When to Use AppVerifier
How to Use AppVerifier
Analyzing AppVerifier Data
Key Points about AppVerifier
Appendix A: Driver Verifier
Appendix B: Using Application Verifier to Test Your Security Features
Appendix C:References

Summary

The Microsoft® Application Verifier (AppVerifier) is a runtime verification tool for unmanaged code that assists in finding subtle programming errors, security issues and limited user account privilege problems that can be difficult to identify with normal application testing techniques.

Introduction

One of the biggest challenges faced by programmers, software architects, testers, and security consultants is to understand the consequences of their applications when deployed into production. Even with access to source code, it is difficult to grasp everything that will occur during execution due to a variety of dependencies (for example. multiple groups contributing to code or leveraging external components). The Microsoft AppVerifier can play a critical role in helping to manage this complexity and the potential side effects of bugs. The AppVerifier assists in finding subtle programming errors, security issues, and user account privilege problems that can be difficult to identify during a typical test pass.

This article will present information about how AppVerifier works and how it can be applied during the Software Development Lifecycle.

What Is AppVerifier?

AppVerifier is a free download specifically designed to detect and help debug memory corruptions, critical security vulnerabilities, and limited user account privilege issues. AppVerifier aids in the creation of reliable and secure applications by monitoring an application's interaction with the Microsoft® Windows® operating system, and profiling its use of objects, the registry, the file system, and Win32 APIs (including heaps, handles, and locks). AppVerifier also includes checks to predict how well the application will perform in non-admin environments.

When used throughout the software development lifecycle, AppVerifier can bring cost benefits to development efforts because it facilitates identifying problems early on when they are easier and cheaper to fix. It also helps to detect errors that may have gone unnoticed and ensures that the final application can be executed in restricted (for example, non-admin) environments (this last point will become more important with Windows Vista applications).

Problems That AppVerifier Identifies

AppVerifier helps to determine:

  • When the application is using APIs correctly:
    • Unsafe TerminateThread APIs.
    • Correct use of Thread Local Storage (TLS) APIs.
    • Correct use of virtual space manipulations (for example, VirtualAlloc, MapViewOfFile).
  • Whether the application is hiding access violations using structured exception handling.
  • Whether the application is attempting to use invalid handles.
  • Whether there are memory corruptions or issues in the heap.
  • Whether the application runs out of memory under low resources.
  • Whether the correct usage of critical sections is occurring.
  • Whether an application running in an administrative environment will run well in an environment with less privilege.
  • Whether there are potential problems when the application is running as a limited user.
  • Whether there are uninitialized variables in future function calls in a thread's context.

AppVerifier Features

AppVerifier was created in 2001 and is leveraged during the logon process. However, with the release of the 3.0 version to the Download Center there have been some changes. The user interface has changed, and there is now an added focus on security, an updated architecture that provides more robust reporting (XML based logs), and flexibility in adding new tests (such as LuaPriv, and there are more to come).

AppVerifier Tests

AppVerifier consists of sets of tests called "verification layers." These can be turned on or off for each application being tested. By expanding the verification layer within the tests area, the specific tests are displayed. To turn on a test for the application, select the check box next to it. To turn on an entire verification layer, such as Basics, select the check box at the top level.

There are four different types of tests that AppVerifier can perform (in version 3.0.28):

  1. Basics consists of tests for:
    • Exceptions — Ensures that the application does not hide access violations using structured exception handling.
    • Handles — Ensures that the application does not attempt to use invalid handles.
    • Heaps — Checks for memory corruption issues in the heap.
    • Locks — Verifies the correct usage for critical sections.
    • Memory — Ensures that APIs for virtual space manipulations are used correctly.
    • Threadpool — Checks for dirty threadpool threads and other threadpool-related issues.
    • TLS — Ensures that Thread Local Storage APIs are used correctly.
  2. Compatibility tests help to identify problems an application may have with the Windows operating system. This is used in the Windows logon process and consists of the following:
    • FilePaths — Identifies applications that use file paths not obtained by using approved APIs.
    • HighVersionLie — Identifies problems for some of the most common application compatibility issues in Windows. Improperly detecting the version of the operating system or using hard-coded version information can cause the application to fail on later operating systems.
    • InteractiveServices — Logs problems if detected on systems vulnerable to security attacks and possibly not running properly on Windows Vista.
    • KernelModeDriverInstall — Logs the installation of a kernel mode driver. When shipping a driver, a developer should be aware that it needs to be tested by the driver verifier and preferably signed by WQHL.
  3. Low resource simulation tries to simulate an environment under low resources, whereby you can define a number (0–100) that indicates the fault probability calls of:
    • Wait (for example. the WaitForXXXX API).
    • Heap_Alloc (Heap Allocation API).
    • Virtual_Alloc (Virtual Memory allocation API).
    • Registry (Registry API).
    • File (File API such as CreateFile).
    • Event (Event API such as CreateEvent).
    • MapView ( MapView API such as CreateMapView).
    • Ole_Alloc (Ole API such as SysAllocString).
  4. LuaPriv contains 31 different tests that can be used in two different scenarios:
    • Predictive — Determines whether an application running with administrative privileges would work if run with less privilege (generally, as a normal user). For example, if the application writes to files that only allow access to administrators, then that application will not be able to write to the same file if run as a non-admin.
    • Diagnostic — When running as a non-admin, identifies potential problems that may already exist with the current run. For example, if the application tries to write to a file that only grants admin access, the application will get an ACCESS_DENIED error.
  5. Miscellaneous consists of tests for:
    • Dangerous APIs — Tracks to see if the application is using the following unsafe actions:
      • Dangerous call to TerminateThread.
      • Potential stack overflow in low memory conditions.
      • Exit process called while multiple threads are still running.
      • LoadLibrary is called during DllMain.
      • FreeLibrary is called during DllMain.
    • Dirty Stacks fills (periodically) the unused portion of the stack with a memory pattern. This can help detect uninitialized variables in future function calls in that thread's context.
    • TimeRollOver forces the GetTickCount and TimeGetTime APIs to roll over faster than they normally would. This allows applications to test their handling of time rollover more easily.
  6. Printing helps find and troubleshoot issues that may result when the print subsystem is called by an application. It targets the two layers of the print subsystem:
    • Print API
    • Print Driver

Properties

There are two sets of properties within AppVerifier. One is based on the test and the other on the application. Each of the properties is defined by the following:

  • Name — Unique name for each of the properties.
  • Type — Boolean, DWORD, String, MultipleString.
  • Value — The element that can be changed and varies based on type.
  • Description — The description explains what the property is.

Editing Properties per Application

For each application the following properties are available to edit:

  • Propagate — Propagate verifier setting from parent process to child processes. False (not checking the box) does not propagate the settings where True (checking the box) does.
  • AutoClr — After a specified image starts to run, the verified image will clear the settings for itself. False (not checking the box) will not do this; True (checking the box) will.
  • AutoDisableStop — AppVerifier will only complain once about an error. If the issue is found again, it will not generate an error. False will generate an error each time the problem is found. True will only generate one error.
  • LoggingWithLocksHeld — The dll load/unload event will be logged. Verifier is doing I/O when the loaderlock is in a hold. This may hang the application. False will not log the event, and True will.

Editing Properties for Tests

Please refer to the Help contents within AppVerifier for the details regarding the properties for specific tests. To edit the properties, you can take one of two approaches:

  • Individual — In the property window, double-click the property you want to edit. This will display a dialog box that allows you to edit the item as well as set it back to the default. Change the entry and click OK.
  • Group — From the Test area, click the test or verification layer you need to edit. Use the right-click options to display a screen of all the properties. Change the properties or reset back to the default and click OK.

Logs

Logs are created when an application is run and errors are generated. Logs are in an XML format and can be viewed using a browser, XML, or an XLST.

When to Use AppVerifier

You should use AppVerifier throughout your Software Development Lifecycle (SDL, also referred to as the Security Development Lifecycle):

  • Requirements Phase — Plan for the use of AppVerifier and schedule its execution and follow up.
  • Design Phase — Define which components (modules, Dlls, or EXEs) will be tested.
  • Implementation Phase — Run AppVerifier on stable builds (from Alpha to RTM) of the different components under development (it is important to test the components individually and collectively).
  • Verification Phase — Testers should execute all of their tests (both manual and automatic) with AppVerifier because this is the first time the application will be pushed to the limits and unexpected behavior and data will be submitted. AppVerifier is also a powerful tool for security consultants doing audits (black box and white box) because it will allow the quick enumeration of real (or potential) attack/exploit vectors.
  • Release Phase — Clients and security consultants can use AppVerifier on the released binaries to identify potential security vulnerabilities.
  • Support and Servicing Phase — Use AppVerifier to ensure that code changes (for example, a bug fix) do not introduce regressions.

How to Use AppVerifier

Installing and Configuring

  1. Install: Download the latest version of AppVerifier from the Microsoft Download Center where you will find setup binaries for the supported architectures: x86, ia64, amd64.

  2. Debugger setup: The application being verified should run under a user-mode debugger or the system should run under a kernel debugger because it will break into a debugger when an error occurs. For more debugger details, see the Help contents.

  3. Settings: AppVerifier cannot be enabled for a running process. You need to create settings as described below and then start the application. The settings are persistent until explicitly deleted: No matter how many times you launch an application it will start with AppVerifier enabled until the settings are deleted.

    Note   AppVerifier is also available within the Visual Studio 2005 Team System.

Using AppVerifier Basics Test

The scenarios below showcase the recommended command line and user interface options. These should be run during all tests that exercise the code to ensure complete coverage. The expectation for these scenarios is that the application does not break into debugger and all tests pass with the same pass rate as when run without AppVerifier enabled.

  1. Enable verifier for the applications you want to test by using:

    From the command line: appverif /verify MyApp.exe From the user interface: a. Add your application by right-clicking within the Applications area and clicking Add Application. b. Note that the Basics in the Tests area are automatically selected for you. c. Click the Save button.

    Aa480483.appverifier_fig1(en-us,MSDN.10).gif

    • Note:
      • /verify will enable the basics tests.
      • If you are testing a DLL application, AppVerifier has to be enabled for the test .exe that is exercising the DLL.
      • Run the verification layers separately. For example, in one session enable all Basics and in another enable all LuaPriv checks.
    1. Run ALL your tests exercising the application.
    2. Analyze any debugger breaks encountered. If a break occurs, you need to understand why and fix it. (The Help contents provide details on the breaks and how to investigate them.)
    3. When finished, delete all settings:

    From the command line: appverif /n MyApp.exe From the user interface: a. Remove your application by right-clicking within the Applications area and clicking Delete Application. b. Click the Save button.

    Aa480483.appverifier_fig2(en-us,MSDN.10).gif

Using AppVerifier Low Resource Simulation (Fault Injection)

The expectation for this scenario is that the application does not break into the debugger. This means that you have no errors that need to be addressed.

The pass rate for the tests may decrease significantly since random fault injections are introduced into the normal operation.

  1. Enable Application Verifier low resource simulation (fault injection) for the applications:

    From the command line: Appverif /verify MyApp.exe /faults From the user interface:
    a. Add your application by right-clicking within the Applications area and clicking Add Application. b. Select Low Resource Simulation from the Tests area. c. Click the Save button.

    Aa480483.appverifier_fig3(en-us,MSDN.10).gif

    Note: If you are testing a DLL, you can apply low resource simulation (fault injection) on a particular DLL instead of the whole process. The command line format would be:

    appverif /verify TARGET [/faults [PROBABILITY [TIMEOUT [DLL …]]]]

    Example:

    appverif /verify mytest.exe /faults 5 1000 d3d9.dll

  2. Run ALL your tests exercising the application.

  3. When finished, delete all settings. Analyze any debugger breaks encountered. If a break occurs, you will need to understand why and fix it.

  4. When finished, delete all settings:

    From the command line: appverif /n MyApp.exe From the user interface: a. Remove your application by right-clicking within the Applications area and clicking Delete Application. b. Click the Save button.

    Aa480483.appverifier_fig3(en-us,MSDN.10).gif

    Note   Running with and without fault injection exercises largely different code paths in an application and therefore both scenarios must be run in order to get the full benefit of AppVerifier.

Analyzing AppVerifier Data

All data created during AppVerifier analysis is stored in the %ALLUSERSPROFILE%\AppVerifierLogs folder in a binary format. These logs can then be converted to XML by using the user interface or command line for further analysis. To view the XML files, use of one of the following techniques:

  • Web Browser — All modern web browsers are able to display XML files in a structured and color-coded format.
  • XSL Transformations — Create an XSLT that converts the raw XML content into reports that contain only information relevant to you.
  • Import into Excel — Import the XML file into Excel and use filters or Pivot Tables to reorganize and analyze the collected data.
  • Import into Database — Save the XML files and import them into, for example, a SQL or Access database.

Key Points about AppVerifier

This section provides information about the most common questions that potential AppVerifier users have:

System Requirements

AppVerifier is designed to test unmanaged applications (for example, non-.NET Framework applications) on Windows Vista, Windows Server 2003, and Windows XP. While running a full page heap, it is recommended to be at least 1 GB.

AppVerifier does not need access to the source code to execute its tests, although the availability of either the application's symbols or debug information will make a dramatic difference in the quality and usefulness of the data collected.

What Will AppVerifier Validate?

AppVerifier is used for testing user-mode applications in Windows Vista, Windows Server 2003, and Windows XP.

  • To test drivers or kernel modules, leverage Driver Verifier. For more information, see Appendix A: Driver Verifier and the knowledge base article How to Use Driver Verifier to Troubleshoot Windows Drivers.
  • To test Windows CE, use the specific Platform AppVerifier included with the Windows CE Test Kit (CETK).
  • It is important to note that AppVerifier will only test the functions that are called during the test execution. This means that if Unit Tests are available (from the target application or module), they should be used to ensure the maximum code coverage — errors will only be detected if the methods that generate requests are invoked.

How Does AppVerifier Work?

AppVerifier works by modifying the unmanaged DLLs Method Tables so that the required checks are performed before the real function is executed (this is also called "Function Hooking"). For example, the address of the Win32 API CreateFileA method is replaced with an internal AppVerifier method that will trigger a series of tests that, when positive, will be logged.

When new processes are started, the use of AppVerifier's Method Table Hooking techniques is controlled by entries made in specific registry keys. If the registry entry exists, then the AppVerifier DLL will be loaded in a newly created process that will handle the Method Table replacements in the existent and subsequently loaded DLLs. Because these hooks are made when the DLL is loaded, it is not possible to use AppVerifier 3.0 on a process that is already running.

The AppVerifier user interface (UI) is used to control the Registry Key settings and to provide information about the existing logs. After the application and tests are set within the UI and the "Save" button is clicked, the Registry settings are made. The application will then need to be restarted, which will start the monitoring. It is important to note that the settings will persist until the application is removed from AppVerifier.

When a problem is identified, a verifier stop will occur. The number provided is used to identify the exact nature and reason for its occurrence. To better understand the stops, refer to the detailed explanations included in the AppVerifier Help file: Appverif.chm.

Page Heap Technical Details

To detect heap corruptions (overflows or underflows), AppVerifier will modify the way memory is allocated by padding the requested memory with either full non-writable pages or with special tags before and after the allocated memory. AppVerifier does this in Windows Vista, Windows Server 2003, and Windows XP by loading Verifier.dll into the process being verified and redirecting some of the Win32 Heap APIs called by the application to corresponding Verifier.dll APIs.

When padding the requested memory with full non-writable pages (the FULL setting is enabled in the page heap properties section and is the default setting), AppVerifier will consume a large amount of virtual memory but has the advantage that heap corruption events are cached in real time when the overflow or underflow occurs. Remember that the memory in this mode will look either like this [AppVerifier Read-Only Heap Page (4k)] [Amount of memory requested by Application under test] or like this [Amount of memory requested by Application under test] [AppVerifier Read-Only Heap Page (4k)].

The heap check will place a guard page at the beginning or end of allocation depending on the Backward property. If Backward is set to False, which is the default, it will place a guard page at the end of the allocation to catch buffer overruns. If it is set to True, the guard page is placed at the beginning of the allocation to catch buffer underruns.

When padding the requested memory with special tags (enabled by clearing the "Full" check box item in the heap properties), AppVerifier will check and alert you when this memory is released. The main problem in using this technique is that there are some cases when the memory corruption will only be detected when the memory is released (the minimum amount of memory block is 8 bytes), so when on a 3-byte variable or a 5-byte overflow occurs it will not be immediately detected.

On an underflow event, an attempt will be made to write to a Read-Only page. This will trigger an exception. Note that this exception can only be caught if the target application is being executed under a debugger. Note that the full page heap mode will also detect these errors because it uses padding+guard pages. The reason you would use light page heap is if your computer cannot tolerate the high memory constraints of full page heap.

For memory intensive applications, or when it is required to use AppVerifier during long periods of time (for example, stress testing), it is better to run normal (light) heap tests instead of full mode due to the performance degradation. However, when you run into an issue, turn on full page heap to investigate further.

Applications that are using custom heaps (a heap that bypasses the operating system's implementation of the heap) might not get the full benefit of using page heap or might even malfunction when it is enabled.

Appendix A: Driver Verifier

Driver Verifier is a tool to detect errors in a driver (kernel mode software) while the driver is running. Typical findings are: pool corruptions (including buffer overrruns), incorrect validation of untrusted data, and incorrect kernel-mode operations. Whenever Driver Verifier finds an issue, it will crash the system in a controlled manner (bugcheck) and a kernel debugger will be needed to understand the issue. Driver Verifier is an operating system component that is available in Windows 2000 and later.

Driver Verifier Usage Scenarios

  1. Install: Driver Verifier settings are made by using Verifier.exe, which is a system tool that is installed by default on any Windows system. To run the user interface, open a Command Prompt window and type verifier.exe or go to Windows\System32\Verifier.exe.
  2. Settings: To start testing with Driver Verifier enabled, make the settings described below and then restart your system. The settings are persistent until explicitly deleted.

Using Driver Verifier

The scenarios below showcase the recommended command line and user interface options. These should be run during all tests that exercise the code to ensure complete coverage. The expectation for these scenarios is that driver does not produce a system crash and that all tests pass with the same pass rate as when without Driver Verifier enabled.

  1. Enable the standard settings for the drivers you want to test and then reboot:

    From the command line: verifier /flags 0x1FB /driver MyDriver.sys From the user interface: a. Click Create Standard Settings on the initial screen. b. Click the Next button.

    Aa480483.appverifier_fig5(en-us,MSDN.10).gif

    c. Click Select driver names from a list.

    Aa480483.appverifier_fig6(en-us,MSDN.10).gif

    d. Select the drivers you want to test and click the Finish button.

  2. Run ALL your tests exercising the drivers.

  3. Analyze any system crashes encountered. Any system crash is a bug found by the verifier and you will need to understand it and fix it.

Using with Driver Verifier Low Resource Simulation (Fault Injection)

The expectation for these scenarios is that driver does not produce a system crash when Driver Verifier is enabled. Note that the pass rate for the tests may decrease significantly since random faults are introduced into the system.

  1. Enable Low Resource Simulation (fault injection) and reboot:

    From the command line: verifier /volatile /flags 7 From the user interface: a. Click Create custom settings [for code developers] on the initial screen and click the Next button.

    Aa480483.appverifier_fig8(en-us,MSDN.10).gif

    b. Select Low resource simulation and click Next.

    Aa480483.appverifier_fig9(en-us,MSDN.10).gif

    c. Ensure the drivers are selected as in the previous scenario and reboot.

  2. Run again ALL the tests exercising the driver.

  3. Analyze any system crashes encountered.

  4. Disable Low Resource Simulation (fault injection) and reboot:

    From the command line: verifier /reset will delete all settings, verifier /volatile /flags 3 will turn off just fault injection. From the user interface: a. Click Delete existing settings and click Finish.

    Aa480483.appverifier_fig10(en-us,MSDN.10).gif

    b. Reboot.

To debug a verifier system crash, start by running the debugger command !analyze. For more information, use the DDK/LDK (Driver Development Kit) Help.

Note   Running with and without fault injection exercises largely different code paths in a driver and therefore both scenarios must be run to get the full benefit of Driver Verifier.

Appendix B: Using Application Verifier to Test Your Security Features

This section written by Lauren Lavoie.

Summary: Describes using the Application Verifier tool to identify common security problems in your project code.

Code reviews and threat models provide excellent static test tools to help find security bugs in the applications you test. AppVerifier can run tests dynamically in the background while you run your regular test suite on your application. AppVerifier monitors the application's behavior by employing the Windows® shim engine to hook application programming interface (API) calls.

The original AppVerifier detected behavior that causes applications to crash. Later additions to the tool enable it to verify more subtle runtime behavior, such as compatibility with Windows, checking for the correct usage of common APIs, and checking essential features related to security.

Sometimes developers unwittingly write code that is susceptible to attack by Trojan Horse programs just by setting security attributes incorrectly or by misusing common API functions. AppVerifier catches and logs situations that may allow unwanted code to be executed so that developers can make their code more secure.

Checking for Proper Use of DACLs

The most common way developers make their code susceptible to attack is by improper use of Discretionary Access Control Lists (DACLs), which specify which groups can access a particular resource and what kind of access they have. A number of APIs that create or access resources take a DACL as one of their parameters. AppVerifier:

  • Intercepts these API calls and checks that the DACL parameter is valid. This is one of the simplest and most effective ways that AppVerifier checks that resources have secure access permissions.
  • Logs errors whenever it encounters a NULL DACL or any DACL that gives unnecessary permissions to all users (for example, allows all users to change access permissions or to delete a resource). In addition, AppVerifier considers it erroneous to allow any user who is not an administrator to change the access permissions or owner of a resource.
  • Checks the access permissions whenever a file is opened (because files created by other processes might not be subject to the above AppVerifier checks when they are created) or moved (because a file's security attributes are not copied when the file is moved).

Checking for Proper Use of CreateProcess

Calls to the CreateProcess API function are subject to attack if parameters are not specified correctly. AppVerifier generates an error if CreateProcess (or other related API functions) are called with a NULL lpApplicationName parameter and an lpCommandLine parameter that contains spaces. For example, it does not allow the following as the command line parameter:

c:\program files\sample.exe –t –g c:\program files\sample\test

Using this command line, an application can inadvertently execute unwanted code if a malicious user installs his program to C:\Program.

Enforcing Safe Use of Socket APIs

AppVerifier checks that applications that invoke socket functions use the SO_EXCLUSIVEADDRUSE flag. Doing so prevents unwanted hosts using SO_REUSEADDR from binding onto a port and hijacking the host.

Obtaining and Using AppVerifier

Remember, you don't need to add any tests to your test suite to take advantage of AppVerifier. It configures Windows to monitor all of your application's activity, all the time, until you tell it to stop. Take a look at AppVerifier Help for details about its tests, and how AppVerifier reports errors. You can get the latest version of AppVerifier from the Microsoft Download Center.

Appendix C: References

AppVerifier locations

The standalone version of Application Verifier is provided as a free download from the Microsoft Download Center, and there is a version integrated into Visual Studio 2005 Team Edition.

AppVerifier Articles

Help contents within the Application Verifier tool (file Appverif.chm)

Michael Howard: Analyzing Your Applications with Windows Application Verifier, December 3, 2003 (AppVerifier v2.5)

Deploying Windows XP - Application Compatibility, November 2005 (AppVerifier v3.0)

Frequently Asked Questions about Windows Application Verifier, (AppVerifier v3.0 )

Testing Applications with AppVerifier, (AppVerifier v3.0)

New Security Features in Visual Studio 2005, information about the subset of Basic checks (heaps, handles, and locks) that are integrated into Visual Studio Team System.

Security Development Lifecycle Articles

Improving Security Across the Software Development Lifecycle

Software Development Life Cycle (SDLC)

The Trustworthy Computing Security Development Lifecycle

Function Hooking and Patching Articles

Detours

From The Code Project:

API hooking revealed

Process-wide API spying - an ultimate hack