Edit

Share via

Custom grain storage

In the tutorial on declarative actor storage, you learned how to allow grains to store their state in an Azure table using one of the built-in storage providers. While Azure is a great place to store your data, many alternatives exist. There are so many that supporting them all isn't feasible. Instead, Orleans is designed to let you easily add support for your preferred storage by writing a custom grain storage provider.

In this tutorial, you'll walk through how to write a simple file-based grain storage provider. A file system isn't the best place to store grain states because it's local, can have issues with file locks, and the last update date isn't sufficient to prevent inconsistency. However, it's an easy example to illustrate the implementation of a grain storage provider.

Get started

An Orleans grain storage provider is a class that implements IGrainStorage, included in the Microsoft.Orleans.Core NuGet package. It also inherits from ILifecycleParticipant<ISiloLifecycle>, allowing you to subscribe to specific events in the silo's lifecycle. Start by creating a class named FileGrainStorage.

using Microsoft.Extensions.Options;
using Orleans.Configuration;
using Orleans.Runtime;
using Orleans.Storage;

namespace GrainStorage;

public sealed class FileGrainStorage : IGrainStorage, ILifecycleParticipant<ISiloLifecycle>
{
    private readonly string _storageName;
    private readonly FileGrainStorageOptions _options;
    private readonly ClusterOptions _clusterOptions;

    public FileGrainStorage(
        string storageName,
        FileGrainStorageOptions options,
        IOptions<ClusterOptions> clusterOptions)
    {
        _storageName = storageName;
        _options = options;
        _clusterOptions = clusterOptions.Value;
    }

    public Task ClearStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        throw new NotImplementedException();
    }

    public Task ReadStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        throw new NotImplementedException();
    }

    public Task WriteStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        throw new NotImplementedException();
    }

    public void Participate(ISiloLifecycle lifecycle) =>
        throw new NotImplementedException();
}

Each method implements the corresponding method in the IGrainStorage interface, accepting a generic type parameter for the underlying state type. The methods are:

The ILifecycleParticipant<TLifecycleObservable>.Participate method subscribes to the silo's lifecycle.

Before starting the implementation, create an options class containing the root directory where grain state files are persisted. Create an options file named FileGrainStorageOptions containing the following:

using Orleans.Storage;

namespace GrainStorage;

public sealed class FileGrainStorageOptions : IStorageProviderSerializerOptions
{
    public required string RootDirectory { get; set; }

    public required IGrainStorageSerializer GrainStorageSerializer { get; set; }
}

With the options class created, explore the constructor parameters of the FileGrainStorage class:

  • storageName: Specifies which grains should use this storage provider, for example, [StorageProvider(ProviderName = "File")].
  • options: The options class just created.
  • clusterOptions: The cluster options used for retrieving the ServiceId.

Initialize the storage

To initialize the storage, subscribe to the ServiceLifecycleStage.ApplicationServices stage with an onStart function. Consider the following ILifecycleParticipant<TLifecycleObservable>.Participate implementation:

public void Participate(ISiloLifecycle lifecycle) =>
    lifecycle.Subscribe(
        observerName: OptionFormattingUtilities.Name<FileGrainStorage>(_storageName),
        stage: ServiceLifecycleStage.ApplicationServices,
        onStart: (ct) =>
        {
            Directory.CreateDirectory(_options.RootDirectory);
            return Task.CompletedTask;
        });

The onStart function conditionally creates the root directory to store grain states if it doesn't already exist.

Also, provide a common function to construct the filename, ensuring uniqueness per service, grain ID, and grain type:

private string GetKeyString(string grainType, GrainId grainId) =>
    $"{_clusterOptions.ServiceId}.{grainId.Key}.{grainType}";

Read state

To read a grain state, get the filename using the GetKeyString function and combine it with the root directory from the _options instance.

public async Task ReadStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var fName = GetKeyString(stateName, grainId);
    var path = Path.Combine(_options.RootDirectory, fName!);
    var fileInfo = new FileInfo(path);
    if (fileInfo is { Exists: false })
    {
        grainState.State = (T)Activator.CreateInstance(typeof(T))!;
        return;
    }

    using var stream = fileInfo.OpenText();
    var storedData = await stream.ReadToEndAsync();
    
    grainState.State = _options.GrainStorageSerializer.Deserialize<T>(new BinaryData(storedData));
    grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
}

Use fileInfo.LastWriteTimeUtc as an ETag, which other functions use for inconsistency checks to prevent data loss.

For deserialization, use the IStorageProviderSerializerOptions.GrainStorageSerializer. This is important for correctly serializing and deserializing the state.

Write state

Writing the state is similar to reading the state.

public async Task WriteStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var storedData = _options.GrainStorageSerializer.Serialize(grainState.State);
    var fName = GetKeyString(stateName, grainId);
    var path = Path.Combine(_options.RootDirectory, fName!);
    var fileInfo = new FileInfo(path);
    if (fileInfo.Exists && fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
    {
        throw new InconsistentStateException($"""
            Version conflict (WriteState): ServiceId={_clusterOptions.ServiceId}
            ProviderName={_storageName} GrainType={typeof(T)}
            GrainReference={grainId}.
            """);
    }

    await File.WriteAllBytesAsync(path, storedData.ToArray());

    fileInfo.Refresh();
    grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
}

Similar to reading state, use the IStorageProviderSerializerOptions.GrainStorageSerializer to write the state. The current ETag checks against the file's last updated UTC time. If the date differs, it means another activation of the same grain changed the state concurrently. In this situation, throw an InconsistentStateException. This results in the current activation being killed to prevent overwriting the state previously saved by the other activated grain.

Clear state

Clearing the state involves deleting the file if it exists.

public Task ClearStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var fName = GetKeyString(stateName, grainId);
    var path = Path.Combine(_options.RootDirectory, fName!);
    var fileInfo = new FileInfo(path);
    if (fileInfo.Exists)
    {
        if (fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
        {
            throw new InconsistentStateException($"""
                Version conflict (ClearState): ServiceId={_clusterOptions.ServiceId}
                ProviderName={_storageName} GrainType={typeof(T)}
                GrainReference={grainId}.
                """);
        }

        grainState.ETag = null;
        grainState.State = (T)Activator.CreateInstance(typeof(T))!;

        fileInfo.Delete();
    }

    return Task.CompletedTask;
}

For the same reason as WriteStateAsync, check for inconsistency. Before deleting the file and resetting the ETag, check if the current ETag matches the last write time UTC.

Put it all together

Next, create a factory that allows scoping the options to the provider name while creating an instance of FileGrainStorage to ease registration with the service collection.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using Orleans.Configuration.Overrides;
using Orleans.Storage;

namespace GrainStorage;

internal static class FileGrainStorageFactory
{
    internal static IGrainStorage Create(
        IServiceProvider services, string name)
    {
        var optionsMonitor =
            services.GetRequiredService<IOptionsMonitor<FileGrainStorageOptions>>();

        return ActivatorUtilities.CreateInstance<FileGrainStorage>(
            services,
            name,
            optionsMonitor.Get(name),
            services.GetProviderClusterOptions(name));
    }
}

Lastly, to register the grain storage, create an extension on ISiloBuilder. This extension internally registers the grain storage as a named service using AddSingletonNamedService, an extension provided by Orleans.Core.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using Orleans.Runtime;
using Orleans.Storage;

namespace GrainStorage;

public static class FileSiloBuilderExtensions
{
    public static ISiloBuilder AddFileGrainStorage(
        this ISiloBuilder builder,
        string providerName,
        Action<FileGrainStorageOptions> options) =>
        builder.ConfigureServices(
            services => services.AddFileGrainStorage(
                providerName, options));

    public static IServiceCollection AddFileGrainStorage(
        this IServiceCollection services,
        string providerName,
        Action<FileGrainStorageOptions> options)
    {
        services.AddOptions<FileGrainStorageOptions>(providerName)
            .Configure(options);

        services.AddTransient<
            IPostConfigureOptions<FileGrainStorageOptions>,
            DefaultStorageProviderSerializerOptionsConfigurator<FileGrainStorageOptions>>();

        return services.AddSingletonNamedService(providerName, FileGrainStorageFactory.Create)
            .AddSingletonNamedService(providerName,
                (p, n) =>
                    (ILifecycleParticipant<ISiloLifecycle>)p.GetRequiredServiceByName<IGrainStorage>(n));
    }
}

The FileGrainStorage implements two interfaces, IGrainStorage and ILifecycleParticipant<ISiloLifecycle>. Therefore, register two named services, one for each interface:

return services.AddSingletonNamedService(providerName, FileGrainStorageFactory.Create)
    .AddSingletonNamedService(providerName,
        (p, n) => (ILifecycleParticipant<ISiloLifecycle>)p.GetRequiredServiceByName<IGrainStorage>(n));

This enables adding the file storage using the extension on ISiloBuilder:

using GrainStorage;
using Microsoft.Extensions.Hosting;

using IHost host = new HostBuilder()
    .UseOrleans(builder =>
    {
        builder.UseLocalhostClustering()
            .AddFileGrainStorage("File", options =>
            {
                string path = Environment.GetFolderPath(
                    Environment.SpecialFolder.ApplicationData);

                options.RootDirectory = Path.Combine(path, "Orleans/GrainState/v1");
            });
    })
    .Build();

await host.RunAsync();

Now you can decorate your grains with the provider [StorageProvider(ProviderName = "File")], and it stores the grain state in the root directory set in the options. Consider the full implementation of FileGrainStorage:

using Microsoft.Extensions.Options;
using Orleans.Configuration;
using Orleans.Runtime;
using Orleans.Storage;

namespace GrainStorage;

public sealed class FileGrainStorage : IGrainStorage, ILifecycleParticipant<ISiloLifecycle>
{
    private readonly string _storageName;
    private readonly FileGrainStorageOptions _options;
    private readonly ClusterOptions _clusterOptions;

    public FileGrainStorage(
        string storageName,
        FileGrainStorageOptions options,
        IOptions<ClusterOptions> clusterOptions)
    {
        _storageName = storageName;
        _options = options;
        _clusterOptions = clusterOptions.Value;
    }

    // <clearstateasync>
    public Task ClearStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var fName = GetKeyString(stateName, grainId);
        var path = Path.Combine(_options.RootDirectory, fName!);
        var fileInfo = new FileInfo(path);
        if (fileInfo.Exists)
        {
            if (fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
            {
                throw new InconsistentStateException($"""
                    Version conflict (ClearState): ServiceId={_clusterOptions.ServiceId}
                    ProviderName={_storageName} GrainType={typeof(T)}
                    GrainReference={grainId}.
                    """);
            }

            grainState.ETag = null;
            grainState.State = (T)Activator.CreateInstance(typeof(T))!;

            fileInfo.Delete();
        }

        return Task.CompletedTask;
    }
    // </clearstateasync>
    // <readstateasync>
    public async Task ReadStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var fName = GetKeyString(stateName, grainId);
        var path = Path.Combine(_options.RootDirectory, fName!);
        var fileInfo = new FileInfo(path);
        if (fileInfo is { Exists: false })
        {
            grainState.State = (T)Activator.CreateInstance(typeof(T))!;
            return;
        }

        using var stream = fileInfo.OpenText();
        var storedData = await stream.ReadToEndAsync();
        
        grainState.State = _options.GrainStorageSerializer.Deserialize<T>(new BinaryData(storedData));
        grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
    }
    // </readstateasync>
    // <writestateasync>
    public async Task WriteStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var storedData = _options.GrainStorageSerializer.Serialize(grainState.State);
        var fName = GetKeyString(stateName, grainId);
        var path = Path.Combine(_options.RootDirectory, fName!);
        var fileInfo = new FileInfo(path);
        if (fileInfo.Exists && fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
        {
            throw new InconsistentStateException($"""
                Version conflict (WriteState): ServiceId={_clusterOptions.ServiceId}
                ProviderName={_storageName} GrainType={typeof(T)}
                GrainReference={grainId}.
                """);
        }

        await File.WriteAllBytesAsync(path, storedData.ToArray());

        fileInfo.Refresh();
        grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
    }
    // </writestateasync>
    // <participate>
    public void Participate(ISiloLifecycle lifecycle) =>
        lifecycle.Subscribe(
            observerName: OptionFormattingUtilities.Name<FileGrainStorage>(_storageName),
            stage: ServiceLifecycleStage.ApplicationServices,
            onStart: (ct) =>
            {
                Directory.CreateDirectory(_options.RootDirectory);
                return Task.CompletedTask;
            });
    // </participate>
    // <getkeystring>
    private string GetKeyString(string grainType, GrainId grainId) =>
        $"{_clusterOptions.ServiceId}.{grainId.Key}.{grainType}";
    // </getkeystring>
}

using System;
using System.Threading.Tasks;
using Orleans;
using Orleans.Storage;
using Orleans.Runtime;

namespace GrainStorage;

public class FileGrainStorage
    : IGrainStorage, ILifecycleParticipant<ISiloLifecycle>
{
    private readonly string _storageName;
    private readonly FileGrainStorageOptions _options;
    private readonly ClusterOptions _clusterOptions;
    private readonly IGrainFactory _grainFactory;
    private readonly ITypeResolver _typeResolver;
    private JsonSerializerSettings _jsonSettings;

    public FileGrainStorage(
        string storageName,
        FileGrainStorageOptions options,
        IOptions<ClusterOptions> clusterOptions,
        IGrainFactory grainFactory,
        ITypeResolver typeResolver)
    {
        _storageName = storageName;
        _options = options;
        _clusterOptions = clusterOptions.Value;
        _grainFactory = grainFactory;
        _typeResolver = typeResolver;
    }

    public Task ClearStateAsync(
        string grainType,
        GrainReference grainReference,
        IGrainState grainState)
    {
        throw new NotImplementedException();
    }

    public Task ReadStateAsync(
        string grainType,
        GrainReference grainReference,
        IGrainState grainState)
    {
        throw new NotImplementedException();
    }

    public Task WriteStateAsync(
        string grainType,
        GrainReference grainReference,
        IGrainState grainState)
    {
        throw new NotImplementedException();
    }

    public void Participate(
        ISiloLifecycle lifecycle)
    {
        throw new NotImplementedException();
    }
}

Before starting the implementation, create an options class containing the root directory where grain state files are stored. Create an options file named FileGrainStorageOptions:

public class FileGrainStorageOptions
{
    public string RootDirectory { get; set; }
}

Create a constructor containing two fields: storageName to specify which grains should use this storage ([StorageProvider(ProviderName = "File")]) and directory, the directory where grain states are saved.

IGrainFactory and ITypeResolver are used in the next section to initialize the storage.

Also, take two options as arguments: your own FileGrainStorageOptions and the ClusterOptions. These are needed for implementing the storage functionalities.

You also need JsonSerializerSettings as you are serializing and deserializing in JSON format.

Important

JSON is an implementation detail. It's up to you to decide which serialization/deserialization protocol fits your application. Another common format is binary.

Initialize the storage

To initialize the storage, register an Init function on the ApplicationServices lifecycle.

public void Participate(ISiloLifecycle lifecycle)
{
    lifecycle.Subscribe(
        OptionFormattingUtilities.Name<FileGrainStorage>(_storageName),
        ServiceLifecycleStage.ApplicationServices,
        Init);
}

The Init function sets the _jsonSettings used to configure the JSON serializer. At the same time, create the folder to store grain states if it doesn't exist yet.

private Task Init(CancellationToken ct)
{
    // Settings could be made configurable from Options.
    _jsonSettings =
        OrleansJsonSerializer.UpdateSerializerSettings(
            OrleansJsonSerializer.GetDefaultSerializerSettings(
                _typeResolver,
                _grainFactory),
            false,
            false,
            null);

    var directory = new System.IO.DirectoryInfo(_rootDirectory);
    if (!directory.Exists)
        directory.Create();

    return Task.CompletedTask;
}

Also, provide a common function to construct the filename, ensuring uniqueness per service, grain ID, and grain type.

private string GetKeyString(string grainType, GrainReference grainReference)
{
    return $"{_clusterOptions.ServiceId}.{grainReference.ToKeyString()}.{grainType}";
}

Read state

To read a grain state, get the filename using the previously defined function and combine it with the root directory from the options.

public async Task ReadStateAsync(
    string grainType,
    GrainReference grainReference,
    IGrainState grainState)
{
    var fName = GetKeyString(grainType, grainReference);
    var path = Path.Combine(_options.RootDirectory, fName);

    var fileInfo = new FileInfo(path);
    if (!fileInfo.Exists)
    {
        grainState.State = Activator.CreateInstance(grainState.State.GetType());
        return;
    }

    using (var stream = fileInfo.OpenText())
    {
        var storedData = await stream.ReadToEndAsync();
        grainState.State = JsonConvert.DeserializeObject(storedData, _jsonSettings);
    }

    grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
}

Use fileInfo.LastWriteTimeUtc as an ETag, which other functions use for inconsistency checks to prevent data loss.

Note that for deserialization, use the _jsonSettings set in the Init function. This is important for correctly serializing/deserializing the state.

Write state

Writing the state is similar to reading the state.

public async Task WriteStateAsync(
    string grainType,
    GrainReference grainReference,
    IGrainState grainState)
{
    var storedData = JsonConvert.SerializeObject(grainState.State, _jsonSettings);

    var fName = GetKeyString(grainType, grainReference);
    var path = Path.Combine(_options.RootDirectory, fName);

    var fileInfo = new FileInfo(path);

    if (fileInfo.Exists && fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
    {
        throw new InconsistentStateException(
            $"Version conflict (WriteState): ServiceId={_clusterOptions.ServiceId} " +
            $"ProviderName={_storageName} GrainType={grainType} " +
            $"GrainReference={grainReference.ToKeyString()}.");
    }

    using (var stream = new StreamWriter(fileInfo.Open(FileMode.Create, FileAccess.Write)))
    {
        await stream.WriteAsync(storedData);
    }

    fileInfo.Refresh();
    grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
}

Similar to reading state, use _jsonSettings to write the state. The current ETag checks against the file's last updated UTC time. If the date differs, it means another activation of the same grain changed the state concurrently. In this situation, throw an InconsistentStateException, which results in the current activation being killed to prevent overwriting the state previously saved by the other activated grain.

Clear state

Clearing the state involves deleting the file if it exists.

public Task ClearStateAsync(
    string grainType,
    GrainReference grainReference,
    IGrainState grainState)
{
    var fName = GetKeyString(grainType, grainReference);
    var path = Path.Combine(_options.RootDirectory, fName);

    var fileInfo = new FileInfo(path);
    if (fileInfo.Exists)
    {
        if (fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
        {
            throw new InconsistentStateException(
                $"Version conflict (ClearState): ServiceId={_clusterOptions.ServiceId} " +
                $"ProviderName={_storageName} GrainType={grainType} " +
                $"GrainReference={grainReference.ToKeyString()}.");
        }

        grainState.ETag = null;
        grainState.State = Activator.CreateInstance(grainState.State.GetType());
        fileInfo.Delete();
    }

    return Task.CompletedTask;
}

For the same reason as WriteStateAsync, check for inconsistency. Before deleting the file and resetting the ETag, check if the current ETag matches the last write time UTC.

Put it all together

Next, create a factory that allows scoping the options to the provider name while creating an instance of FileGrainStorage to ease registration with the service collection.

public static class FileGrainStorageFactory
{
    internal static IGrainStorage Create(
        IServiceProvider services, string name)
    {
        IOptionsSnapshot<FileGrainStorageOptions> optionsSnapshot =
            services.GetRequiredService<IOptionsSnapshot<FileGrainStorageOptions>>();

        return ActivatorUtilities.CreateInstance<FileGrainStorage>(
            services,
            name,
            optionsSnapshot.Get(name),
            services.GetProviderClusterOptions(name));
    }
}

Lastly, to register the grain storage, create an extension on ISiloHostBuilder. This extension internally registers the grain storage as a named service using .AddSingletonNamedService(...), an extension provided by Orleans.Core.

public static class FileSiloBuilderExtensions
{
    public static ISiloHostBuilder AddFileGrainStorage(
        this ISiloHostBuilder builder,
        string providerName,
        Action<FileGrainStorageOptions> options)
    {
        return builder.ConfigureServices(
            services => services.AddFileGrainStorage(providerName, options));
    }

    public static IServiceCollection AddFileGrainStorage(
        this IServiceCollection services,
        string providerName,
        Action<FileGrainStorageOptions> options)
    {
        services.AddOptions<FileGrainStorageOptions>(providerName).Configure(options);

        return services.AddSingletonNamedService(providerName, FileGrainStorageFactory.Create)
            .AddSingletonNamedService(
                providerName,
                (s, n) => (ILifecycleParticipant<ISiloLifecycle>)s.GetRequiredServiceByName<IGrainStorage>(n));
    }
}

The FileGrainStorage implements two interfaces, IGrainStorage and ILifecycleParticipant<ISiloLifecycle>. Therefore, register two named services, one for each interface:

return services.AddSingletonNamedService(providerName, FileGrainStorageFactory.Create)
    .AddSingletonNamedService(
        providerName,
        (s, n) => (ILifecycleParticipant<ISiloLifecycle>)s.GetRequiredServiceByName<IGrainStorage>(n));

This enables adding the file storage using the extension on ISiloHostBuilder:

var silo = new HostBuilder()
    .UseOrleans(builder =>
    {
        builder.UseLocalhostClustering()
            .AddFileGrainStorage("File", opts =>
            {
                opts.RootDirectory = "C:/TestFiles";
            });
    })
    .Build();

Now you can decorate your grains with the provider [StorageProvider(ProviderName = "File")], and it stores the grain state in the root directory set in the options.