Scheduling overview

Two forms of scheduling in Orleans are relevant to grains:

1. Request scheduling: Scheduling incoming grain calls for execution according to rules discussed in Request scheduling.
2. Task scheduling: Scheduling synchronous blocks of code to execute in a single-threaded manner.

All grain code executes on the grain's task scheduler, meaning requests also execute on the grain's task scheduler. Even if request scheduling rules allow multiple requests to execute concurrently, they won't execute in parallel because the grain's task scheduler always executes tasks one by one and never executes multiple tasks in parallel.

Task scheduling

To better understand scheduling, consider the following grain, MyGrain. It has a method called DelayExecution() that logs a message, waits some time, then logs another message before returning.

public interface IMyGrain : IGrain
{
    Task DelayExecution();
}

public class MyGrain : Grain, IMyGrain
{
    private readonly ILogger<MyGrain> _logger;

    public MyGrain(ILogger<MyGrain> logger) => _logger = logger;

    public async Task DelayExecution()
    {
        _logger.LogInformation("Executing first task");

        await Task.Delay(1_000);

        _logger.LogInformation("Executing second task");
    }
}

When this method executes, the method body executes in two parts:

1. The first _logger.LogInformation(...) call and the call to Task.Delay(1_000).
2. The second _logger.LogInformation(...) call.

The second task isn't scheduled on the grain's task scheduler until the Task.Delay(1_000) call completes. At that point, it schedules the continuation of the grain method.

Here's a graphical representation of how a request is scheduled and executed as two tasks:

The description above isn't specific to Orleans; it describes how task scheduling works in .NET. The C# compiler converts asynchronous methods into an asynchronous state machine, and execution progresses through this state machine in discrete steps. Each step schedules on the current TaskScheduler (accessed via TaskScheduler.Current, defaulting to TaskScheduler.Default) or the current SynchronizationContext. If a TaskScheduler is used, each step in the method represents a Task instance passed to that TaskScheduler. Therefore, a Task in .NET can represent two conceptual things:

1. An asynchronous operation that can be awaited. The execution of the DelayExecution() method above is represented by a Task that can be awaited.
2. A synchronous block of work. Each stage within the DelayExecution() method above is represented by a Task.

When TaskScheduler.Default is used, continuations schedule directly onto the .NET ThreadPool and aren't wrapped in a Task object. The wrapping of continuations in Task instances occurs transparently, so developers rarely need to be aware of these implementation details.

Task scheduling in Orleans

Each grain activation has its own TaskScheduler instance responsible for enforcing the single-threaded execution model of grains. Internally, this TaskScheduler is implemented via ActivationTaskScheduler and WorkItemGroup. WorkItemGroup keeps enqueued tasks in a Queue<T> (where T is internally a Task) and implements IThreadPoolWorkItem. To execute each currently enqueued Task, WorkItemGroup schedules itself on the .NET ThreadPool. When the .NET ThreadPool invokes the WorkItemGroup's IThreadPoolWorkItem.Execute() method, the WorkItemGroup executes the enqueued Task instances one by one.

Each grain has a scheduler that executes by scheduling itself on the .NET ThreadPool:

Each scheduler contains a queue of tasks:

The .NET ThreadPool executes each work item enqueued to it. This includes grain schedulers as well as other work items, such as those scheduled via Task.Run(...):

Note

A grain's scheduler can only execute on one thread at a time, but it doesn't always execute on the same thread. The .NET ThreadPool is free to use a different thread each time the grain's scheduler executes. The grain's scheduler ensures it only executes on one thread at a time, implementing the single-threaded execution model of grains.