October 2011
Volume 26 Number 10
Asynchronous Programming - Easier Asynchronous Programming with the New Visual Studio Async CTP
By Eric Lippert | October 2011
Imagine what the world would be like if people worked the same way as computer programs:
void ServeBreakfast(Customer diner)
{
var order = ObtainOrder(diner);
var ingredients = ObtainIngredients(order);
var recipe = ObtainRecipe(order);
var meal = recipe.Prepare(ingredients);
diner.Give(meal);
}
Each subroutine can, of course, be broken down further; preparing the meal might involve heating pans, cooking omelets and toasting bread. Were humans to perform these sorts of tasks like typical computer programs, we’d carefully write down everything as sequences of hierarchical tasks in a checklist and obsessively ensure that each job was complete before embarking on the next.
A subroutine-based approach seems reasonable—you can’t cook the eggs before you get the order—but in fact it both wastes time and makes the application appear unresponsive. It wastes time because you want the bread to be toasting while the eggs are frying, not after the eggs are done and getting cold. It appears unresponsive because if another customer arrives while the current order is still cooking, you want to be taking his order, not making him wait at the door until the current customer has been served her breakfast. A server slavishly following a checklist does not have any ability to respond in a timely manner to unexpected events.
Solution One: Hire More Staff by Making More Threads
Making someone’s breakfast is a whimsical example, but the reality, of course, is anything but. Every time you transfer control into a long-running subroutine on the UI thread, the UI becomes completely unresponsive until the subroutine finishes. How could it be otherwise? Applications respond to UI events by running code on the UI thread, and that thread is obsessively busy doing something else. Only when every job on its list is done will it get around to dealing with the queued-up commands of the frustrated user. The usual solution to this problem is to use concurrencyto do two or more things “at the same time.” (If the two threads are on two independent processors, they might truly be running at the same time. In a world with more threads than processors to dedicate to them, the OS will simulate at-the-same-time concurrency by periodically scheduling a time slice for each thread to control a processor.)
One concurrent solution might be to create a thread pool and assign each new client a specific thread to handle its requests. In our analogy, you could hire a group of servers. When a new diner comes in, an idle server is assigned to the new diner. Each server then independently does the work of taking the order, finding the ingredients, cooking the food and serving it.
The difficulty with this approach is that UI events typically arrive on the same thread and expect to be fully serviced on that thread. Most UI components create requests on the UI thread, and expect to be communicated with only on that thread. Dedicating a new thread to each UI-related task is unlikely to work well.
To address this problem you could have a single foreground thread listening to UI events that does nothing but “take orders” and farm them out to one or more background worker threads. In this analogy, there’s only one server who interacts with the customers, and a kitchen full of cooks who actually do the requested work. The UI thread and the worker threads are then responsible for coordinating their communications. The cooks never talk directly to the diners, but somehow the food gets served anyway.
This certainly solves the “responding to UI events in a timely manner” problem, but it doesn’t resolve the lack of efficiency; the code running on the worker thread is still waiting synchronously for the eggs to cook fully before the bread goes in the toaster. That problem could be solved in turn by adding even more concurrency: You could have two cooks per order, one for the eggs and one for the toast. But that might get pretty expensive. Just how many cooks are you going to need, and what happens when they have to coordinate their work?
Concurrency of this sort introduces many well-known difficulties. First, threads are notoriously heavyweight; a thread by default consumes a million bytes of virtual memory for its stack and many other system resources. Second, UI objects are often “affinitized” to the UI thread and can’t be called from worker threads; the worker thread and the UI thread must come to some complex arrangement whereby the UI thread can send necessary information from the UI elements over to the worker, and the worker can send updates back to the UI thread, rather than to the UI elements directly. Such arrangements are difficult to code and prone to race conditions, deadlocks and other threading problems. Third, many of the pleasant fictions that we all rely upon in the single-threaded world—such as reads and writes of memory happening in a predictable and consistent sequence—are no longer reliable. This leads to the worst kinds of difficult-to-reproduce bugs.
It just seems wrong to have to use the big hammer of thread-based concurrency to build simple programs that remain responsive and run efficiently. Somehow real people manage to solve complex problems while remaining responsive to events. In the real world you don’t have to allocate one waiter per table or two cooks per order to serve dozens of customer requests that are all pending at the same time. Solving the problem with threading makes for too many cooks. There’s got to be a better solution that doesn’t involve so much concurrency.
Solution Two: Develop Attention Deficit Disorder with DoEvents
A common non-concurrent “solution” to the problem of UI unresponsiveness during long-running operations is to liberally sprinkle the magic words Application.DoEvents around a program until the problem goes away. Though this is certainly a pragmatic solution, it’s not a very well-engineered one:
void ServeBreakfast(Customer diner)
{
var order = ObtainOrder(diner);
Application.DoEvents();
var ingredients = ObtainIngredients(order);
Application.DoEvents();
var recipe = ObtainRecipe(order);
Application.DoEvents();
var meal = recipe.Prepare(ingredients);
Application.DoEvents();
diner.Give(meal);
}
Basically, using DoEvents means “see if anything interesting happened while I was busy doing that last thing. If something happened that I need to respond to, remember what I was doing just now, deal with the new situation, and then come back to where I left off.” It makes your program behave like it has attention deficit disorder: anything new that comes along gets attention right away. That sounds like a plausible solution to improve responsiveness—and sometimes even works—but there are a number of problems with this approach.
First, DoEvents works best when the delay is caused by a loop that has to execute many times, but each individual loop execution is short. By checking for pending events every few times through the loop, you can maintain responsiveness even if the whole loop takes a long time to run. However, that pattern is usually not the cause of a responsiveness problem. More often the problem is caused by one inherently long-running operation taking a lot of time, such as attempting to synchronously access a file over a high-latency network. Perhaps in our example the long-running task is in preparing the meal, and there’s no place to put the DoEvents that helps. Or perhaps there is a place where DoEvents would help, but it’s in a method you don’t have the source code for.
Second, calling DoEvents causes the program to attempt to fully service all the more recent events before finishing work associated with earlier events. Imagine if no one could get his meal until after every customer who came in got his meal! If more and more customers keep arriving, the first customer might never get his meal, resulting in starvation. In fact, it could happen that no customers get their meals. The completion of work associated with earlier events can be pushed off arbitrarily far into the future as servicing newer events continues to interrupt the work being done for earlier events.
Third, DoEvents poses the very real danger of unexpected reentrancy. That is, while serving one customer you check to see if there have been any recent interesting UI events and accidentally start serving the same diner again, even though he’s already being served. Most developers don’t design their code to detect this kind of reentrancy; it’s possible to end up in some very strange program states indeed when an algorithm never intended to be recursive ends up calling itself unexpectedly via DoEvents.
In short, DoEvents should be used only to fix a responsiveness problem in the most trivial cases; it’s not a good solution for managing UI responsiveness in complex programs.
Solution Three: Turn Your Checklist Inside-out with Callbacks
The non-concurrent nature of the DoEvents technique is attractive, but clearly not quite the right solution for a complex program. A better idea is to break down the items on the checklist into a series of short tasks, each of which can be completed rapidly enough that the application can appear to be responsive to events.
That idea is nothing new; dividing a complex problem into small parts is why we have subroutines in the first place. The interesting twist is that instead of rigidly running down a checklist to determine what has already been done and what needs to be done next, and only returning control to the caller when everything is completed, each new task is given the list of work that must come after it. The work that must come after a particular task is finished is called the “continuation” of the task.
When a task has finished, it can look at the continuation and finish it off right there. Or it might schedule the continuation to run later. If the continuation requires information computed by the previous task, the previous task can pass that information along as an argument to the call that invokes the continuation.
With this approach, the total body of work is essentially broken up into little pieces that can each be executed rapidly. The system appears responsive because pending events can be detected and handled between the executions of any two of the small pieces of work. But because any activities associated with those new events can also be broken down into small parts and queued up to execute later, we don’t have the “starvation” problem whereby new tasks prevent old tasks from completing. New long-running tasks are not dealt with immediately, but they are queued up for eventual processing.
The idea is great, but it’s not at all clear how to implement such a solution. The essential difficulty is determining how to tell each small unit of work what its continuation is; that is, what work needs to come next.
In traditional asynchronous code, this is typically done by registering a “callback” function. Let’s suppose we have an asynchronous version of “Prepare” that takes a callback function that says what to do next—namely, serve the meal:
void ServeBreakfast(Diner diner)
{
var order = ObtainOrder(diner);
var ingredients = ObtainIngredients(order);
var recipe = ObtainRecipe(order);
recipe.PrepareAsync(ingredients, meal =>
{
diner.Give(meal);
});
}
Now ServeBreakfast returns immediately after PrepareAsync returns; whatever code called ServeBreakfast is then free to service other events that occur. PrepareAsync does no “real” work itself; rather, it quickly does whatever is necessary to ensure that the meal will be prepared in the future. Moreover, PrepareAsync also ensures that the callback method will be invoked with the prepared meal as its argument at some time after the meal preparation task is completed. Thus, the diner will eventually be served, though she might have to wait briefly if there’s an event that requires attention between the end of the preparation and the serving of the meal.
Note that none of this necessarily involves a second thread. Perhaps PrepareAsync causes the meal preparation work to be done on a separate thread, or perhaps it causes a series of short tasks associated with meal preparation to be queued up on the UI thread to be executed later. It really doesn’t matter; all we know is that PrepareAsync somehow guarantees two things: that the meal will be prepared in a manner that doesn’t block the UI thread with a high-latency operation, and that the callback will somehow be invoked after the work of preparing the requested meal is done.
But suppose any of the methods for obtaining the order, obtaining the ingredients, obtaining the recipe or preparing the meal might be the one that’s slowing down the UI. We could solve this larger problem if we had an asynchronous version of each of these methods. What would the resulting program look like? Remember, each method must be given a callback that tells it what to do when the unit of work is completed:
void ServeBreakfast(Diner diner)
{
ObtainOrderAsync(diner, order =>
{
ObtainIngredientsAsync(order, ingredients =>
{
ObtainRecipeAsync(order, recipe =>
{
recipe.PrepareAsync(ingredients, meal =>
{
diner.Give(meal);
})})})});
}
This might seem like an awful mess, but it’s nothing compared to how bad real programs get when they’re rewritten using callback-based asynchrony. Think about how you might deal with making a loop asynchronous, or how you’d deal with exceptions, try-finally blocks or other non-trivial forms of control flow. You end up essentially turning your program inside-out; the code now emphasizes how all the callbacks are wired together, and not what the logical workflow of the program should be.
Solution Four: Make the Compiler Solve the Problem with Task-Based Asynchrony
Callback-based asynchrony does keep the UI thread responsive and minimize time wasted by synchronously waiting for long-running work to complete. But the cure seems worse than the disease. The price you pay for responsiveness and performance is that you have to write code that emphasizes how the mechanisms of the asynchrony work while obscuring the meaning and purpose of the code.
The upcoming versions of C# and Visual Basic instead allow you to write code that emphasizes its meaning and purpose, while giving enough hints to the compilers to build the necessary mechanisms for you behind the scenes. The solution has two parts: one in the type system, and one in the language.
The CLR 4 release defined the type Task<T>—the workhorse type of the Task Parallel Library (TPL)—to represent the concept of “some work that’s going to produce a result of type T in the future.” The concept of “work that will complete in the future but returns no result” is represented by the non-generic Task type.
Precisely how the result of type T is going to be produced in the future is an implementation detail of a particular task; the work might be farmed out to another machine entirely, to another process on this machine, to another thread, or perhaps the work is simply to read a previously cached result that can be accessed cheaply from the current thread. TPL tasks are typically farmed out to worker threads from a thread pool in the current process, but that implementation detail is not fundamental to the Task<T> type; rather, a Task<T> can represent any high-latency operation that produces a T.
The language half of the solution is the new await keyword. A regular method call means “remember what you’re doing, run this method until it’s completely finished, and then pick up where you left off, now knowing the result of the method.” An await expression, in contrast, means “evaluate this expression to obtain an object representing work that will in the future produce a result. Sign up the remainder of the current method as the callback associated with the continuation of that task. Once the task is produced and the callback is signed up, immediately return control to my caller.”
Our little example rewritten in the new style reads much more nicely:
async void ServeBreakfast(Diner diner)
{
var order = await ObtainOrderAsync(diner);
var ingredients = await ObtainIngredientsAsync(order);
var recipe = await ObtainRecipeAsync(order);
var meal = await recipe.PrepareAsync(ingredients);
diner.Give(meal);
}
In this sketch, each asynchronous version returns a Task<Order>, Task<List<Ingredient>> and so on. Every time an await is encountered, the currently executing method signs up the rest of the method as the thing to do when the current task is complete, and then immediately returns. Somehow each task will complete itself—either by being scheduled to run as an event on the current thread, or because it used an I/O completion thread or worker thread—and will then cause its continuation to “pick up where it left off” in executing the rest of the method.
Note that the method is now marked with the new async keyword; this is simply an indicator to the compiler that lets it know that in the context of this method, the keyword await is to be treated as a point where the workflow returns control to its caller and picks up again when the associated task is finished. Note also that the examples I’ve shown in this article use C# code; Visual Basic will have a similar feature with similar syntax. The design of these features in C# and Visual Basic was heavily influenced by F# asynchronous workflows, a feature that F# has had for some time.
Where to Learn More
This brief introduction merely motivates and then scratches the surface of the new asynchrony feature in C# and Visual Basic. For a more detailed explanation of how it works behind the scenes, and how to reason about the performance characteristics of asynchronous code, see the companion articles in this issue by my colleagues Mads Torgersen and Stephen Toub.
To get your hands on a preview release of this feature, along with samples, white papers and a community forum for questions, discussions and constructive feedback, please go to msdn.com/async. These language features and the libraries that support them are still in development; the design team would love to have as much of your feedback as possible.
Eric Lippert is a principal developer on the C# Compiler team at Microsoft.
Thanks to the following technical experts for reviewing this article: *Mads Torgersen, *Stephen Toub *and *Lucian Wischik