November 2014
Volume 29 Number 11
Async Programming : Unit Testing Asynchronous Code: Three Solutions for Better Tests
Asynchronous programming has become more and more important during the last decade. Whether for CPU-based parallelism or IO-based concurrency, developers are employing asynchrony to help make the most of the resources available and, ultimately, do more with less. More responsive client applications and more scalable server applications are all within reach.
Software developers have learned a lot of design patterns for effectively building synchronous functionality, but best practices for designing asynchronous software are relatively new, though the support provided by programming languages and libraries for parallel and concurrent programming has dramatically improved with the release of the Microsoft .NET Framework 4 and 4.5. While there’s already quite a lot of good advice for using the new techniques (see “Best Practices in Asynchronous Programming” at bit.ly/1ulDCiI and “Talk: Async Best Practices” at bit.ly/1DsFuMi), best practices for designing the internal and external APIs for applications and libraries with language features like async and await and the Task Parallel Library (TPL) are still unknown to many developers.
This gap affects not just the performance and reliability of the applications and libraries such developers are building, but also the testability of their solutions, as many of the best practices that enable the creation of robust asynchronous designs enable easier unit testing, as well.
With such best practices in mind, this article will present ways to design and refactor code for better testability, and demonstrate how this will influence the tests. The solutions are applicable to code that takes advantage of async and await, as well as code based on lower-level multithreading mechanisms from earlier frameworks and libraries. And, in the process, the solutions will not only be better factored for testing, they’ll be more easily and more efficiently consumable by users of the developed code.
The team I work with is developing software for medical X-ray devices. In this domain it’s critical our unit-test coverage is always at a high level. Recently, a developer asked me, “You’re always pushing us to write unit tests for all our code. But how can I write reasonable unit tests when my code starts another thread or is using a timer that later starts a thread and runs it several times?”
That is a meaningful question. Suppose I have this code to be tested:
public void StartAsynchronousOperation()
{
Message = "Init";
Task.Run(() =>
{
Thread.Sleep(1900);
Message += " Work";
});
}
public string Message { get; private set; }
My first attempt to write a test for that code was not very promising:
[Test]
public void FragileAndSlowTest()
{
var sut = new SystemUnderTest();
// Operation to be tested will run in another thread.
sut. StartAsynchronousOperation();
// Bad idea: Hoping that the other thread finishes execution after 2 seconds.
Thread.Sleep(2000);
// Assert outcome of the thread.
Assert.AreEqual("Init Work", sut.Message);
}
I expect unit tests to run fast and deliver predictable results, but the test I wrote was fragile and slow. The StartAsynchronousOperation method kicks off the operation to be tested in another thread, and the test should check the outcome of the operation. The time it takes to start a new thread or to initialize an existing thread sitting in the thread pool and to execute the operation isn’t predictable, because it depends on other processes running on the test machine. The test may fail from time to time when the sleep is too short and the asynchronous operation hasn’t yet finished. I’m between the devil and the deep blue sea: Either I try to keep the waiting time as short as possible, with the risk of a fragile test, or I increase the sleep time to make the test more robust, but slow down the test even more.
The problem is similar when I want to test code that uses a timer:
private System.Threading.Timer timer;
private readonly Object lockObject = new Object();
public void StartRecurring()
{
Message = "Init";
// Set up timer with 1 sec delay and 1 sec interval.
timer = new Timer(o => { lock(lockObject){ Message += " Poll";} }, null,
new TimeSpan(0, 0, 0, 1), new TimeSpan(0, 0, 0, 1));
}
public string Message { get; private set; }
And this test is likely to have the same problems:
[Test]
public void FragileAndSlowTestWithTimer()
{
var sut = new SystemUnderTest();
// Execute code to set up timer with 1 sec delay and interval.
sut.StartRecurring();
// Bad idea: Wait for timer to trigger three times.
Thread.Sleep(3100);
// Assert outcome.
Assert.AreEqual("Init Poll Poll Poll", sut.Message);
}
When I test a reasonably complex operation with several different code branches, I end up with a huge number of separate tests. My test suites get slower and slower with each new test. The maintenance costs for these test suites increase, because I have to spend time investigating the sporadic failures. Moreover, slow test suites tend to be executed only infrequently and therefore have fewer benefits. At some point, I’d probably stop executing these slow and intermittently failing tests altogether.
The two kinds of tests shown earlier are also unable to detect when the operation throws an exception. Because the operation runs in a different thread, exceptions aren’t propagated to the test runner thread. This limits the ability of tests to check the correct error behavior of the code under test.
I’m going to present three general solutions for avoiding slow, fragile unit tests by improving the design of the tested code, and I’ll show how this enables unit tests to check exceptions. Each solution has advantages, as well as disadvantages or limitations. At the end, I’ll give some recommendations regarding which solution to choose for different situations.
This article is about unit testing. Typically, unit tests are testing code in isolation from other parts of the system. One of those other parts of the system is the OS multithreading capability. Standard library classes and methods are used to schedule asynchronous work, but the multithreading aspect should be excluded in unit tests, which should concentrate on the functionality that runs asynchronously.
Unit tests for asynchronous code make sense when the code contains chunks of functionality running in one thread, and the unit tests should verify that the chunks are working as expected. When the unit tests have shown that the functionality is correct, it makes sense to use additional test strategies to uncover concurrency problems. There are several approaches for testing and analyzing multithreaded code to find these kinds of problems (see, for example, “Tools and Techniques to Identify Concurrency Issues,” at bit.ly/1tVjpll). Stress tests, for example, can put a whole system or a large part of the system under load. These strategies are sensible to complement unit tests, but are out of scope for this article. The solutions in this article will show how to exclude the multithreading parts while testing the functionality in isolation with unit tests.
Solution 1: Separate Functionality from Multithreading
The simplest solution for unit testing the functionality involved in asynchronous operations is to separate that functionality from multithreading. Gerard Mezaros has described this approach in the Humble Object pattern in his book, “xUnit Test Patterns” (Addison-Wesley, 2007). The functionality to be tested is extracted into a separate new class and the multithreading part stays within the Humble Object, which calls the new class (see Figure 1).
.png "The Humble Object Pattern")
Figure 1 The Humble Object Pattern
The following code shows the extracted functionality after refactoring, which is purely synchronous code:
public class Functionality
{
public void Init()
{
Message = "Init";
}
public void Do()
{
Message += " Work";
}
public string Message { get; private set; }
}
Before refactoring, the functionality was mixed with asynchronous code, but I’ve moved it to the Functionality class. This class can now be tested via simple unit tests because it doesn't contain any multithreading anymore. Note that such refactoring is important in general, not just for unit testing: components shouldn’t expose asynchronous wrappers for inherently synchronous operations and should instead leave it up to the caller to determine whether to offload the invocation of that operation. In the case of a unit test, I choose not to, but the consuming application might decide to do so, for reasons of either responsiveness or parallel execution. For more information, see Stephen Toub’s blog post, “Should I Expose Asynchronous Wrappers for Synchronous Methods?” (bit.ly/1shQPfn).
In a lightweight variation of this pattern, certain private methods of the SystemUnderTest class can be made public to allow tests to call these methods directly. In this case, no additional class needs to be created for testing the functionality without multithreading.
Separating the functionality via the Humble Object pattern is simple and can be done not only for code that immediately schedules asynchronous work once, but also for code that uses timers. In that case the timer handling is kept in the Humble Object and the recurring operation is moved to the Functionality class or a public method. An advantage of this solution is that tests can directly check exceptions thrown by the code under test. The Humble Object pattern can be applied regardless of the techniques used to schedule asynchronous work. The drawbacks of this solution are that the code in the Humble Object itself isn’t tested and that the code under test has to be modified.
Solution 2: Synchronize Tests
If the test is able to detect the completion of the operation under test, which runs asynchronously, it can avoid the two disadvantages, fragility and slowness. Though the test runs multithreaded code, it can be reliable and fast when the test synchronizes with the operation scheduled by the code under test. The test can concentrate on the functionality while the negative effects of the asynchronous execution are minimized.
In the best case, the method under test returns an instance of a type that will be signaled when the operation has completed. The Task type, which has been available in the .NET Framework since version 4, meets this need nicely, and the async/await feature available since the .NET Framework 4.5 makes it easy to compose Tasks:
public async Task DoWorkAsync()
{
Message = "Init";
await Task.Run( async() =>
{
await Task.Delay(1900);
Message += " Work";
});
}
public string Message { get; private set; }
This refactoring represents a general best practice that helps in both the unit testing case and in general consumption of the exposed asynchronous functionality. By returning a Task that represents the asynchronous operation, a consumer of the code is able to easily determine when the asynchronous operation has completed, whether it failed with an exception, and whether it returned a result.
This makes unit testing an asynchronous method as simple as unit testing a synchronous method. It’s now easy for the test to synchronize with the code under test simply by invoking the target method and waiting for the returned Task to complete. This waiting can be done synchronously (blocking the calling thread) via the Task Wait methods, or it can be done asynchronously (using continuations to avoid blocking the calling thread) with the await keyword, before checking the outcome of the asynchronous operation (see Figure 2).
.jpg "Synchronization via Async and Await")
Figure 2 Synchronization via Async and Await
In order to use await in a unit test method, the test itself has to be declared with async in its signature. No sleep statement is needed anymore:
[Test]
public async Task SynchronizeTestWithCodeViaAwait()
{
var sut = new SystemUnderTest();
// Schedule operation to run asynchronously and wait until it is finished.
await sut.StartAsync();
// Assert outcome of the operation.
Assert.AreEqual("Init Work", sut.Message);
}
Luckily, the latest versions of the major unit test frameworks—MSTest, xUnit.net and NUnit—support the async and await tests (see Stephen Cleary’s blog at bit.ly/1x18mta). Their test runners can cope with async Task tests and await the completion of the thread before they start to evaluate the assert statements. If the test runner of the unit testing framework can’t cope with async Task test method signatures, the test can at least call the Wait method on the Task returned from the system under test.
In addition, the timer-based functionality can be enhanced with the help of the TaskCompletionSource class (see details in the code download). The test can then await the completion of specific recurring operations:
[Test]
public async Task SynchronizeTestWithRecurringOperationViaAwait()
{
var sut = new SystemUnderTest();
// Execute code to set up timer with 1 sec delay and interval.
var firstNotification = sut.StartRecurring();
// Wait that operation has finished two times.
var secondNotification = await firstNotification.GetNext();
await secondNotification.GetNext();
// Assert outcome.
Assert.AreEqual("Init Poll Poll", sut.Message);
}
Unfortunately, sometimes the code under test can’t use async and await, such as when you’re testing code that has already shipped and for breaking-change reasons the signature of the method being tested can’t be changed. In situations like this, the synchronization has to be implemented with other techniques. Synchronization can be realized if the class under test invokes an event or calls a dependent object when the operation is completed. The following example demonstrates how to implement the test when a dependent object is called:
private readonly ISomeInterface dependent;
public void StartAsynchronousOperation()
{
Task.Run(()=>
{
Message += " Work";
// The asynchronous operation is finished.
dependent.DoMore()
});
}
An additional example for event-based synchronization is in the code download.
The test can now be synchronized with the asynchronous operation when the dependent object is replaced by a stub while testing (see Figure 3).
.png "Synchronization via a Dependent Object Stub")
Figure 3 Synchronization via a Dependent Object Stub
The test has to equip the stub with a thread-safe notification mechanism, because the stub code runs in another thread. In the following test code example, a ManualResetEventSlim is used and the stub is generated with the RhinoMocks mocking framework:
// Setup
var finishedEvent = new ManualResetEventSlim();
var dependentStub = MockRepository.GenerateStub<ISomeInterface>();
dependentStub.Stub(x => x.DoMore()).
WhenCalled(x => finishedEvent.Set());
var sut = new SystemUnderTest(dependentStub);
The test can now execute the asynchronous operation and can wait for the notification:
// Schedule operation to run asynchronously.
sut.StartAsynchronousOperation();
// Wait for operation to be finished.
finishedEvent.Wait();
// Assert outcome of operation.
Assert.AreEqual("Init Work", sut.Message);
This solution to synchronize the test with the tested threads can be applied to code with certain characteristics: The code under test has a notification mechanism like async and await or a plain event, or the code calls a dependent object.
A big advantage of the async and await synchronization is that it’s able to propagate any kind of result back to the calling client. A special kind of result is an exception. So the test can handle exceptions explicitly. The other synchronization mechanisms can only recognize failures indirectly via the defective outcome.
Code with timer-based functionality can utilize async/await, events or calls to dependent objects to allow tests to synchronize with the timer operations. Every time the recurring operation finishes, the test is notified and can check the outcome (see examples in the code download).
Unfortunately, timers make unit tests slow even when you use a notification. The recurring operation you want to test generally starts only after a certain delay. The test will be slowed down and take at least the time of the delay. This is an additional disadvantage on top of the notification prerequisite.
Now I’ll look at a solution that avoids some of the limitations of the two previous ones.
Solution 3: Test in One Thread
For this solution, the code under test has to be prepared in a way that the test can later directly trigger the execution of the operations on the same thread as test itself. This is a translation of the approach of the jMock team for Java (see “Testing Multithreaded Code” at www.jmock.org/threads.html).
The system under test in the following example uses an injected task scheduler object to schedule asynchronous work. To demonstrate the capabilities of the third solution, I added a second operation that will be started when the first operation is finished:
private readonly TaskScheduler taskScheduler;
public void StartAsynchronousOperation()
{
Message = "Init";
Task task1 = Task.Factory.StartNew(()=>{Message += " Work1";},
CancellationToken.None,
TaskCreationOptions.None,
taskScheduler);
task1.ContinueWith(((t)=>{Message += " Work2";}, taskScheduler);
}
The system under test is modified to use a separate TaskScheduler. During the test, the “normal” TaskScheduler is replaced by a DeterministicTaskScheduler, which allows starting the asynchronous operations synchronously (see Figure 4).
.png "Using a Separate TaskScheduler in SystemUnderTest")
Figure 4 Using a Separate TaskScheduler in SystemUnderTest
The following test can execute the scheduled operations in the same thread as the test itself. The test injects the DeterministicTaskScheduler into the code under test. The DeterministicTaskScheduler doesn’t immediately spawn a new thread, but only queues scheduled tasks. In the next statement the RunTasksUntilIdle method synchronously executes the two operations:
[Test]
public void TestCodeSynchronously()
{
var dts = new DeterministicTaskScheduler();
var sut = new SystemUnderTest(dts);
// Execute code to schedule first operation and return immediately.
sut.StartAsynchronousOperation();
// Execute all operations on the current thread.
dts.RunTasksUntilIdle();
// Assert outcome of the two operations.
Assert.AreEqual("Init Work1 Work2", sut.Message);
}
The DeterministicTaskScheduler overrides TaskScheduler methods to provide the scheduling functionality and adds among others the RunTasksUntilIdle method specifically for testing (see the code download for DeterministicTaskScheduler implementation details). As in synchronous unit testing, stubs can be used to concentrate on just a single unit of functionality at a time.
Code that uses timers is problematic not only because tests are fragile and slow. Unit tests get more complicated when the code uses a timer that’s not running on a worker thread. In the .NET framework class library, there are timers specifically designed to be used in UI applications, like System.Windows.Forms.Timer for Windows Forms or System.Windows.Threading.DispatcherTimer for Windows Presentation Foundation (WPF) applications (see “Comparing the Timer Classes in the .NET Framework Class Library” at bit.ly/1r0SVic). These use the UI message queue, which isn’t directly available during unit testing. The test shown at the beginning of this article will not work for these timers. The test has to spin up the message pump, for example by using WPF DispatcherFrame (see the example in the code download). To keep the unit tests simple and clear when you’re deploying UI-based timers, you have to replace these timers during testing. I’m introducing an interface for timers to enable replacing the “real” timers with an implementation specifically for testing. I do this also for “thread”-based timers like System.Timers.Timer or System.Threading.Timer, because I can then improve the unit tests in all cases. The system under test has to be modified to use this ITimer interface:
private readonly ITimer timer;
private readonly Object lockObject = new Object();
public void StartRecurring()
{
Message = "Init";
// Set up timer with 1 sec delay and 1 sec interval.
timer.StartTimer(() => { lock(lockObject){Message +=
" Poll";} }, new TimeSpan(0,0,0,1));
}
By introducing the ITimer interface, I can replace the timer behavior during testing, as shown in Figure 5.
.png "Use ITimer in SystemUnderTest")
Figure 5 Use ITimer in SystemUnderTest
The extra effort of defining the interface ITimer pays off because a unit test checking the outcome of the initialization and the recurring operation can now run very quickly and reliably within milliseconds:
[Test]
public void VeryFastAndReliableTestWithTimer()
{
var dt = new DeterministicTimer();
var sut = new SystemUnderTest(dt);
// Execute code that sets up a timer 1 sec delay and 1 sec interval.
sut.StartRecurring();
// Tell timer that some time has elapsed.
dt.ElapseSeconds(3);
// Assert that outcome of three executions of the recurring operation is OK.
Assert.AreEqual("Init Poll Poll Poll", sut.Message);
}
The DeterministicTimer is specifically written for testing purposes. It allows the test to control the point in time when the timer action is executed, without waiting. The action is executed in the same thread as the test itself (see the code download for DeterministicTimer implementation details). For the execution of the tested code in a “non-testing” context, I have to implement an ITimer adapter for an existing timer. The code download contains examples of adapters for several of the framework class library timers. The ITimer interface can be tailored to the needs of the concrete situation and may only contain a subset of the full functionality of specific timers.
Testing asynchronous code with a DeterministicTaskScheduler or a DeterministicTimer allows you to easily switch off multithreading during testing. The functionality is executed on the same thread as the test itself. The interoperation of initialization code and asynchronous code is kept and can be tested. A test of this kind, for example, can check the correct time values used to initialize a timer. Exceptions are forwarded to tests, so they can directly check the error behavior of the code.
Wrapping Up
Effective unit testing of asynchronous code has three main benefits: Maintenance costs for tests are reduced; tests run faster; and the risk of not executing the tests anymore is minimized. The solutions presented in this article can help you reach this goal.
The first solution, separating the functionality from the asynchronous aspects of a program via a Humble Object is the most generic. It’s applicable for all situations, regardless of how threads are started. I recommend using this solution for very complex asynchronous scenarios, complex functionality or a combination of both. It’s a good example of the separation of concerns design principle (see bit.ly/1lB8iHD).
The second solution, which synchronizes the test with the completion of the tested thread, can be applied when the code under test provides a synchronization mechanism such as async and await. This solution makes sense when the notification mechanism prerequisites are fulfilled anyhow. If possible, use the elegant async and await synchronization when non-timer threads are started, because exceptions are propagated to the test. Tests with timers can use await, events or calls to dependent objects. These tests can be slow when timers have long delays or intervals.
The third solution uses the DeterministicTaskScheduler and the DeterministicTimer, thus avoiding most limitations and disadvantages of the other solutions. It requires some effort to prepare the code under test, but high unit test code coverage can be reached. The tests for code with timers can be executed very fast without the wait for delay and interval times. Also, exceptions are propagated to the tests. So this solution will lead to robust, fast and elegant unit test suites combined with high code coverage.
These three solutions can help software developers avoid the pitfalls of unit testing asynchronous code. They can be used to create fast and robust unit test suites and cover a wide range of parallel programming techniques.
Sven Grand is a quality engineering software architect for the diagnostic X-ray business unit of Philips Healthcare. He got “test infected” years ago, when he heard for the first time about test-driven development at a Microsoft software conference in 2001. Reach him at sven.grand@philips.com.
Thanks to the following technical experts for reviewing this article: Stephen Cleary, James McCaffrey, Henning Pohl and Stephen Toub