Introduction to TestApi – Part 5: Managed Code Fault Injection APIs

项目
11/25/2009

Series Index

+++

Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.

There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:

Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.

The FI terminology is as follows:

AUT (application under test) – this is the tested application, in which faults are being injected;
Fault Rule – The fault rule is a central construct in an FI test that determines WHEN faults get triggered and WHAT TYPES of faults get triggered. A fault rule consists of:
- a Method Signature, determining the method where the fault will be injected;
- a Fault Condition, determining when the specific fault should be triggered (e.g. every N^th call)
- a Fault -- Determines the type of fault (e.g. throwing an exception, returning a specific value, etc.) that occurs when the fault condition is met.
Fault Session – The fault session is a collection of fault rules that are applied to a given AUT.

TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.

Sample AUT

Following is the code of a trivial AUT that we will use for demonstration purposes:

 //
// This is a sample application used for demonstration purposes.
//
using System;

class MyApplication
{
    static void Main(string[] args)
    {
        int a = 2;
        int b = 3;

        for (int i = 0; i < 10; i++)
        {
            Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b));
        }
    }

    private static int Sum(int a, int b)
    {
        return a + b;
    }
}

The result of running this application is of course:

> MyApplication.exe 0) 2 + 3 = 5 1) 2 + 3 = 5 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5

A Simple Fault Injection Test

Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:

 // 
// Simple fault injection test
//
using System;
using System.Diagnostics;
using Microsoft.Test.FaultInjection;

public class FaultInjectionTest 
{ 
    public static void Main() 
    { 
        // 
        // Set up a fault rule to return –1000 the second time Sum is called.
        //
        string method = "MyApplication.Sum(int,int)";
        ICondition condition = BuiltInConditions.TriggerOnNthCall(2);
        IFault fault = BuiltInFaults.ReturnValueFault(-1000);
        FaultRule rule = new FaultRule(method, condition, fault);            

        //
        // Establish a session, injecting the faults defined by the fault rule(s)
        //
        FaultSession session = new FaultSession(rule);
        ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); 

        //
        // Launch the target process and observe faults
        //
        Process p = Process.Start(psi);        
        p.WaitForExit();
    }
}

Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:

> FaultInjectionTest.exe 0) 2 + 3 = 5 1) 2 + 3 = -1000 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5

As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.

Under The Covers

Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.

Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.

The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.

In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.

I have attached the sample above, which should get you up and running with fault injection.

FaultInjectionSample.zip

Comments

Anonymous
February 24, 2010
I am trying to use the FaultInjection libraries to simulate data access failures. Is this proper method signature to get the ExecuteNonQuery() method on DbCommand? Dim method = "System.Data.Common.DbCommand.ExecuteNonQuery()" Thanks.
Anonymous
February 24, 2010
Yep, this should work. Let me know if you have trouble and we will help.
Anonymous
March 01, 2010
I had success when using the fault injection libraries on some sample code I had written, but now that I'm trying to use it against our intended target code it doesn't seem to be working. I want to use the ReturnValueFault on an internal static method which returns an int. The method is within an internal sealed class with the SecuritySafeCritical attribute set. I'm creating the fault rule like this: FaultRule rule = new FaultRule("static FullNamespace.ClassName.GetDeviceId(uint, IntPtr, IntPtr, IntPtr)", BuiltInConditions.TriggerOnEveryCall, BuiltInFaults.ReturnValueFault(1000)); I also tried prepending out to IntPtr as these are specified as out parameters in the method signature. I didn't have success either way. Am I doing something wrong here? My other thought is that this class and method are in a library/dll, and not within the exe code itself which is the process I'm starting a fault injection session for. Any help is greatly appreciated. So far I've been very happy with the low cost of entry for using this API and the good documentation.
Anonymous
March 01, 2010
Hi David, can you try to replace "IntPtr" with "System.IntPtr" in the parameters list of the method? the data type of the parameter need to be fully qualified name too. let me know if it still doesn't work.
Anonymous
March 03, 2010
I tried System.IntPtr and had no luck here either. Is it an issue if the binary I am trying to inject into is signed?
Anonymous
March 04, 2010
no, the issue is not related to the sign. i noticed that you mention about 'out parameter'. please make sure that 'out' is also part of the method signature. i.e your method signature should be: static FullNamespace.ClassName.GetDeviceId(uint, out System.IntPtr,out System.IntPtr,out System.IntPtr) i wrote a dummy app with the same method signature above, and the fault injection api works for me without problem. if this still doesn't work for you. most likely reason here is that the method signature still not correct somehow, i.e it does not match what clr expecting in the runtime). a few thing you can duble check:

make sure the method got invoked.
make sure the method signature is correct or no typo (i often see people has typo in the method signature by mistakes).
in the %temp% folder, there should have some log, check if there are some error.

Anonymous
March 05, 2010
I have tried with the out parameter and also it did not work. I coulnd't find any logs in %temp% folder. I realized today that the AUT is a native program that runs the code from the managed assemblies by hosting a CLR runtime. Is this my problem, is there a way to still use TestApi fault injection or will I have to use some native fault injection library instead?
Anonymous
March 05, 2010
The comment has been removed
Anonymous
March 05, 2010
Darn, this would have been the perfect solution. Thanks for your help!
Anonymous
March 09, 2010
I am just getting around to responding to Ivan's response to my original comment. Ivan, I am unable to get ExecuteNonQuery() to throw an exception for me. Here is all my code from within a (slightly verbose) unit test: Dim method = "System.Data.Common.DbCommand.ExecuteNonQuery()" 'Dim condition = BuiltInConditions.TriggerOnNthCall(2) Dim condition = BuiltInConditions.TriggerOnEveryCall Const ExceptionMessage As String = "This is a fault-injected exception." Dim fault = BuiltInFaults.ThrowExceptionFault(New DataException(ExceptionMessage)) Dim faultRule = New FaultRule(method, condition, fault) Dim s = New FaultSession(faultRule) Dim rp = TestDataFactory.CreateYrtReinsurancePolicy(PolicyNumberFactory.CreateUnique()) Dim isExceptionThrown As Boolean Try rp.Save() Catch ex As DataException If ex.Message.Equals(ExceptionMessage) Then isExceptionThrown = True End If End Try Assert.IsTrue(isExceptionThrown) Note that I gave up on the NthCall and went with EveryCall just to see if I could get it to work. Any ideas?
Anonymous
March 16, 2010
The comment has been removed
Anonymous
March 18, 2010
The problem with that workflow is that I would need to write a process wrapper for each and every unit test that I write. I think my solution file will get out of control quickly. Given that, I think that I will wait for your private version. Thanks for the clarification. I just didn't infer that launching an additional process was necessary.
Anonymous
May 02, 2010
Hi, When i try to add the FaultInjectionEngine.dll as a reference to my code in VisualC#, it is giving an error which says, "Please make sure that the file is accessible and that it is a valid assembly or COM object". Please help me on this.
Anonymous
May 03, 2010
In your VS project, you should add reference to TestApiCore.dll, not faultinjectionengine.dll.
Anonymous
May 04, 2010
Yah did the same, Still the same error crops in.
Anonymous
May 05, 2010
hmm, that's strange. if you haven't, can you try to create a brand new project and add testapicore.dll? what's the version of your VS, OS, 32bit or 64 bit? also check my blog which has step by step instructions. you can see if you missed any important one. http://blogs.msdn.com/billliu/default.aspx
Anonymous
October 21, 2010
Hi I am planning to use FI API's to introduce faults in to my application hosted on IIS 7.0. Can you please guide me on this.. ? Thanks
Anonymous
October 24, 2010
Re: fault injection in IIS 7.0: Check out Bill's recent post demonstrating that: blogs.msdn.com/.../using-fault-injection-api-for-web-application-or-windows-service.aspx
Anonymous
April 24, 2011
Hi, I am new to this, I copied the sample solution given here and tried on visual studio 2010 . It simply isn't working .. I mean the fault injection test is not injecting the fault at all. Please help.
Anonymous
April 25, 2011
Hi , when i try to run the same sample project on VS 2010 (after conversion) on winXP(SP3) the fault injection simply doesnt work. Please help. I have tried the sample code on VS 2008 on a windows 7. It does work there though I follow the same steps.
Anonymous
June 10, 2012
how to create custom fault and custom condition? please help
Anonymous
June 11, 2012
for vs2010 or .net 4.0 application, you need to enable one env variable. see my blog for steps: blogs.msdn.com/.../if-it-still-not-working.aspx
Anonymous
June 11, 2012
the following code snipet to show how to custom the condition. the condition is to trigger fault if the current machine is management server (regkey settings): [Serializable] class MyCondition : ICondition { public bool Trigger(IRuntimeContext context) { RegistryKey masterKey = Registry.LocalMachine.CreateSubKey("SOFTWARE\Microsoft\FaultInjection"); string reg = ""; if (masterKey == null) { Console.WriteLine("Null Masterkey!"); } else { reg = masterKey.GetValue("ServerType").ToString(); Console.WriteLine("MyKey = {0}", masterKey.GetValue("ServerType")); } masterKey.Close(); return reg.Equals("Management", StringComparison.OrdinalIgnoreCase); } }
Anonymous
January 10, 2013
Wait a minute. I thought you said "It uses the .NET Profiler API to dynamically instrument the binaries as they are running so that the binaries are only altered in memory and not on the hard disk. " But from the examples it seems you always have launch the application under test itself using the test apis/exe but can't actually target an already launched application and inject code into it, even though that's what you claimed.
Anonymous
January 16, 2013
The comment has been removed
Anonymous
February 08, 2013
Thank you for sharing this very nice post, please keep continue the sharing of this types of information. Here we are waiting for more
Anonymous
February 10, 2013
@Performance Injector: Thanks for the nice words! :) We are looking into extending the TestApi facilities further.
Anonymous
February 12, 2013
i hate this it didnt give me what i need
Anonymous
February 12, 2013
@none What didn't work? Can I help?
Anonymous
September 22, 2013
what if Class A implement Interface B, with method C. should I injection to specify to inject "A.C" or "B.C"?

通过