Workflow performance tips: long-running custom activities
Custom activities that perform long-running work can affect workflow performance in unexpected ways. Understanding WF's threading model and scheduler can help activity authors make informed decisions about their code. This post intends to shed some light on the negative performance impact when a custom activity becomes blocking.
Let's start by creating a new workflow console application. Make sure the target framework is set to .Net 4.0. Add a sequence activity to the new workflow and create a variable in that sequence called startTime of type DateTime.
We will be using this variable as a timer to tell us how long the workflow took to execute. To do this, add an Assign activity that assigns the value DateTime.Now to the startTime variable. Then add a WriteLine activity underneath that with the text:
"Execution took " & (DateTime.Now - startTime).TotalSeconds & " seconds"
Now our workflow looks lke this:
In between the Assign and the WriteLine, add a ParallelForEach activity. The Foreach line should look like "ForEach item in {1,2,3,4,5}". This is just a shorthand way of getting this to loop five times. Inside the ParallelForEach, place a sequence activity. In the sequence activity add another WriteLine that prints the branch number with the text "Finished branch " & item.
If you run this workflow now, you should see the branches print in reverse order (5,4,3,2,1). It would be helpful for this experiment to see when the branch starts as well, so we'll copy/paste the WriteLine in the inner sequence and change the text from Finished to Started.
Obviously, in between the start and finished activities is where we're going to do our long-running work. At this point, we'll create a new custom activity. Add a new item to the project and pick CodeActivity under Workflow. Call it SleepActivity. In place of real long-running work, we'll just put a Thread.Sleep() in there since it will have the same effect. Change the custom activity's code to look like this:
public sealed class SleepActivity : CodeActivity
{
public InArgument<TimeSpan>Duration { get; set; }
protected override void Execute(CodeActivityContext context)
{
Thread.Sleep(context.GetValue<TimeSpan>(Duration));
}
}
Press F6 or Ctrl+B to build the project and add the new activity to the toolbox. Then insert it into the inner sequence in our workflow and set the Duration to five seconds.
The workflow is now ready to run. When you run it, you'll see output similar to the following:
Started branch 5
Finished branch 5
Started branch 4
Finished branch 4
Started branch 3
Finished branch 3
Started branch 2
Finished branch 2
Started branch 1
Finished branch 1
Execution took 25.0215019 seconds
So the question is, why is it called ParallelForEach if it's not actually parallel?
The reason for this is that a workflow instance is run on a single thread. The WF runtime schedules work on that thread using a queue-like structure. When the ParallelForEach activity executes, it simply schedules its child activities with the runtime. Because the activities do blocking, synchronous work, they block the thread the workflow is running on. Other effects include hampering the ability for the WF runtime to schedule high priority things such as a cancellation. The cancellation would only be pushed to the front of the queue and would not execute until the current work item is finished.
The point is, if you have long running work to do inside a custom activity, do it asynchronously!
Let's try this by creating a new CodeActivity called AsyncSleepActivity. Once the activity is added to the project, change the code to the following:
public sealed class AsyncSleepActivity : AsyncCodeActivity
{
public InArgument<TimeSpan> Duration { get; set; }
protected override IAsyncResult BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, object state)
{
TimeSpan ts = context.GetValue<TimeSpan>(Duration);
Func<TimeSpan, object> f = new Func<TimeSpan, object>(Sleep);
context.UserState = f;
return f.BeginInvoke(ts, callback, state);
}
private object Sleep(TimeSpan ts)
{
Thread.Sleep(ts);
return null;
}
protected override void EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
{
Func<TimeSpan, object> f = (Func<TimeSpan, object>)context.UserState;
f.EndInvoke(result);
}
}
Here we're inheriting from AsyncCodeActivity instead of CodeActivity and that forces us to override the BeginExecute and EndExecute methods. Press F6 again to build the project and make the AsyncSleepActivity available in the toolbox. Then swap out the old SleepActivity in our workflow for the new activity. Set the Duration to five seconds again and run the program.
Started branch 5
Started branch 4
Started branch 3
Started branch 2
Started branch 1
Finished branch 5
Finished branch 4
Finished branch 3
Finished branch 2
Finished branch 1
Execution took 8.2018201 seconds
The results you see on your own machine may be different than this. The key here is that even though all the activities were scheduled at the same time, they did not finish at the same time. This actually is not the WF runtime's fault. The machine I'm running on has two cores. The number of cores has an effect on the minimum number of threads in the thread pool. In my case, the first two activities got thread pool threads at the same time. The next three got a new thread from the thread pool as they were created in one second intervals. Hence the eight second execution time.
This points out another problem. We may be performing asynchronous work, but we're still blocking a thread. This may be OK for certain circumstances. The workflow thread is allowed to continue doing its job and the .Net ThreadPool will adjust itself accordingly to optimize the number of threads. Keep in mind that an AsyncCodeActivity also creates a no-persist zone while executing.
If we're waiting on input, we may want to allow the workflow to go idle and persist or unload. We also don't want to block a thread as it can have an effect on throughput. The best way to handle this case is to use a bookmark. Bookmarks require some kind of outside interference, so we have to modify the main program a bit. First, we'll switch to WorkflowApplication instead of WorkflowInvoker. WorkflowApplication will allow us to resume a bookmark. We will also make the handling of the timers external to the activity. When a Timer elapses, the Elapsed event handler has to be able to resume the bookmark. Therefore, it needs the WorkflowApplication object and the bookmark name. Change the Program.cs file in the project to look like this:
using System;
using System.Activities;
using System.Threading;
using System.Timers;
namespace WorkflowConsoleApplication1
{
class Program
{
private static WorkflowApplication wfApp = null;
static void Main(string[] args)
{
ManualResetEvent mre = new ManualResetEvent(false);
wfApp = new WorkflowApplication(new Workflow1());
wfApp.Completed = new Action<WorkflowApplicationCompletedEventArgs>(
(WorkflowApplicationCompletedEventArgs e) =>
{
mre.Set();
});
wfApp.Run();
mre.WaitOne();
}
internal static void StartTimer(TimeSpan duration, string bookmarkName)
{
System.Timers.Timer t = new System.Timers.Timer(duration.TotalMilliseconds);
t.AutoReset = false;
t.Elapsed += ((object sender, ElapsedEventArgs e) =>
{
wfApp.ResumeBookmark(bookmarkName, null);
});
t.Start();
}
}
}
Add a new code file to the project called BookmarkSleepActivity. Paste the following code:
using System;
using System.Activities;
namespace WorkflowConsoleApplication1
{
public sealed class BookmarkSleepActivity : NativeActivity
{
public InArgument<TimeSpan> Duration { get; set; }
public InArgument<string> BookmarkName { get; set; }
protected override bool CanInduceIdle { get { return true; } }
protected override void Execute(NativeActivityContext context)
{
Bookmark bookmark = context.CreateBookmark(
context.GetValue<string>(BookmarkName),
new BookmarkCallback(TimerExpired));
Program.StartTimer(context.GetValue<TimeSpan>(Duration),
bookmark.Name);
}
private void TimerExpired(NativeActivityContext context, Bookmark bookmark,
object obj)
{
Console.WriteLine("Bookmark {0} completed", bookmark.Name);
}
}
}
Inside the Execute method of the NativeActivity, we create a bookmark with a name specified by an argument. Then we call Program.StartTimer with the bookmark name so that when the timer elapses, it can resume the bookmark. The callback method for the bookmark doesn't have to do anything special. Notice that the CanInduceIdle property is overridden. This is required to be set to true since a bookmark is used.
Press F6 to build the project and add the new activity to the toolbox. Then replace the AsyncSleepActivity with the new BookmarkSleepActivity in the inner sequence in our workflow. When you execute this workflow you should see the following output.
Started branch 5
Started branch 4
Started branch 3
Started branch 2
Started branch 1
Bookmark Bookmark5 completed
Finished branch 5
Bookmark Bookmark4 completed
Finished branch 4
Bookmark Bookmark3 completed
Finished branch 3
Bookmark Bookmark2 completed
Finished branch 2
Bookmark Bookmark1 completed
Finished branch 1
Execution took 5.0255025 seconds
Finally we have the desired result. When writing custom activities that do long running work, it is clearly bad practice to block the workflow thread. But whether you use AsyncCodeActivity or NativeActivity and bookmarks depends on your situation. If your activity does CPU-intensive work it's best to use AsyncCodeActivity as you probably want to block the thread pool thread anyway. If you're calling a synchronous API, it may also be best to use AsyncCodeActivity as you'll want to have the no-persist zone anyway. But for other things like waiting for input, NativeActivity and bookmarks are the way to go.
Also note that while this could be considered equivalent to a Delay activity, the Delay activity is more advanced to handle more difficult situations. For instance, creating a new Timer object per workflow instance can be costly, especially if the workflow is unloaded. Also, durable timers have to be worked into the mix somehow.