Create Protobuf messages for .NET apps

Article
09/27/2024

Note

This isn't the latest version of this article. For the current release, see the .NET 9 version of this article.

Warning

This version of ASP.NET Core is no longer supported. For more information, see .NET and .NET Core Support Policy. For the current release, see the .NET 8 version of this article.

Important

This information relates to a pre-release product that may be substantially modified before it's commercially released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

For the current release, see the .NET 9 version of this article.

By James Newton-King and Mark Rendle

gRPC uses Protobuf as its Interface Definition Language (IDL). Protobuf IDL is a language neutral format for specifying the messages sent and received by gRPC services. Protobuf messages are defined in .proto files. This document explains how Protobuf concepts map to .NET.

Protobuf messages

Messages are the main data transfer object in Protobuf. They are conceptually similar to .NET classes.

syntax = "proto3";

option csharp_namespace = "Contoso.Messages";

message Person {
    int32 id = 1;
    string first_name = 2;
    string last_name = 3;
}

The preceding message definition specifies three fields as name-value pairs. Like properties on .NET types, each field has a name and a type. The field type can be a Protobuf scalar value type, for example int32, or another message.

The Protobuf style guide recommends using underscore_separated_names for field names. New Protobuf messages created for .NET apps should follow the Protobuf style guidelines. .NET tooling automatically generates .NET types that use .NET naming standards. For example, a first_name Protobuf field generates a FirstName .NET property.

In addition to a name, each field in the message definition has a unique number. Field numbers are used to identify fields when the message is serialized to Protobuf. Serializing a small number is faster than serializing the entire field name. Because field numbers identify a field it is important to take care when changing them. For more information about changing Protobuf messages see Versioning gRPC services.

When an app is built the Protobuf tooling generates .NET types from .proto files. The Person message generates a .NET class:

public class Person
{
    public int Id { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

For more information about Protobuf messages see the Protobuf language guide.

Scalar Value Types

Protobuf supports a range of native scalar value types. The following table lists them all with their equivalent C# type:

Protobuf type	C# type
`double`	`double`
`float`	`float`
`int32`	`int`
`int64`	`long`
`uint32`	`uint`
`uint64`	`ulong`
`sint32`	`int`
`sint64`	`long`
`fixed32`	`uint`
`fixed64`	`ulong`
`sfixed32`	`int`
`sfixed64`	`long`
`bool`	`bool`
`string`	`string`
`bytes`	`ByteString`

Scalar values always have a default value and can't be set to null. This constraint includes string and ByteString which are C# classes. string defaults to an empty string value and ByteString defaults to an empty bytes value. Attempting to set them to null throws an error.

Nullable wrapper types can be used to support null values.

Dates and times

The native scalar types don't provide for date and time values, equivalent to .NET's DateTimeOffset, DateTime, and TimeSpan. These types can be specified by using some of Protobuf's Well-Known Types extensions. These extensions provide code generation and runtime support for complex field types across the supported platforms.

The following table shows the date and time types:

.NET type	Protobuf Well-Known Type
`DateTimeOffset`	`google.protobuf.Timestamp`
`DateTime`	`google.protobuf.Timestamp`
`TimeSpan`	`google.protobuf.Duration`

syntax = "proto3";

import "google/protobuf/duration.proto";  
import "google/protobuf/timestamp.proto";

message Meeting {
    string subject = 1;
    google.protobuf.Timestamp start = 2;
    google.protobuf.Duration duration = 3;
}

The generated properties in the C# class aren't the .NET date and time types. The properties use the Timestamp and Duration classes in the Google.Protobuf.WellKnownTypes namespace. These classes provide methods for converting to and from DateTimeOffset, DateTime, and TimeSpan.

// Create Timestamp and Duration from .NET DateTimeOffset and TimeSpan.
var meeting = new Meeting
{
    Time = Timestamp.FromDateTimeOffset(meetingTime), // also FromDateTime()
    Duration = Duration.FromTimeSpan(meetingLength)
};

// Convert Timestamp and Duration to .NET DateTimeOffset and TimeSpan.
var time = meeting.Time.ToDateTimeOffset();
var duration = meeting.Duration?.ToTimeSpan();

Note

The Timestamp type works with UTC times. DateTimeOffset values always have an offset of zero, and the DateTime.Kind property is always DateTimeKind.Utc.

Nullable types

The Protobuf code generation for C# uses the native types, such as int for int32. So the values are always included and can't be null.

For values that require explicit null, such as using int? in C# code, Protobuf's Well-Known Types include wrappers that are compiled to nullable C# types. To use them, import wrappers.proto into your .proto file, like the following code:

syntax = "proto3";

import "google/protobuf/wrappers.proto";

message Person {
    // ...
    google.protobuf.Int32Value age = 5;
}

wrappers.proto types aren't exposed in generated properties. Protobuf automatically maps them to appropriate .NET nullable types in C# messages. For example, a google.protobuf.Int32Value field generates an int? property. Reference type properties like string and ByteString are unchanged except null can be assigned to them without error.

The following table shows the complete list of wrapper types with their equivalent C# type:

C# type	Well-Known Type wrapper
`bool?`	`google.protobuf.BoolValue`
`double?`	`google.protobuf.DoubleValue`
`float?`	`google.protobuf.FloatValue`
`int?`	`google.protobuf.Int32Value`
`long?`	`google.protobuf.Int64Value`
`uint?`	`google.protobuf.UInt32Value`
`ulong?`	`google.protobuf.UInt64Value`
`string`	`google.protobuf.StringValue`
`ByteString`	`google.protobuf.BytesValue`

Bytes

Binary payloads are supported in Protobuf with the bytes scalar value type. A generated property in C# uses ByteString as the property type.

Use ByteString.CopyFrom(byte[] data) to create a new instance from a byte array:

var data = await File.ReadAllBytesAsync(path);

var payload = new PayloadResponse();
payload.Data = ByteString.CopyFrom(data);

ByteString data is accessed directly using ByteString.Span or ByteString.Memory. Or call ByteString.ToByteArray() to convert an instance back into a byte array:

var payload = await client.GetPayload(new PayloadRequest());

await File.WriteAllBytesAsync(path, payload.Data.ToByteArray());

Decimals

Protobuf doesn't natively support the .NET decimal type, just double and float. There's an ongoing discussion in the Protobuf project about the possibility of adding a standard decimal type to the Well-Known Types, with platform support for languages and frameworks that support it. Nothing has been implemented yet.

It's possible to create a message definition to represent the decimal type that works for safe serialization between .NET clients and servers. But developers on other platforms would have to understand the format being used and implement their own handling for it.

Creating a custom decimal type for Protobuf

package CustomTypes;

// Example: 12345.6789 -> { units = 12345, nanos = 678900000 }
message DecimalValue {

    // Whole units part of the amount
    int64 units = 1;

    // Nano units of the amount (10^-9)
    // Must be same sign as units
    sfixed32 nanos = 2;
}

The nanos field represents values from 0.999_999_999 to -0.999_999_999. For example, the decimal value 1.5m would be represented as { units = 1, nanos = 500_000_000 }. This is why the nanos field in this example uses the sfixed32 type, which encodes more efficiently than int32 for larger values. If the units field is negative, the nanos field should also be negative.

Note

Additional algorithms are available for encoding decimal values as byte strings. The algorithm used by DecimalValue:

Is easy to understand.
Isn't affected by big-endian or little-endian on different platforms.
Supports decimal numbers ranging from positive 9,223,372,036,854,775,807.999999999 to negative 9,223,372,036,854,775,808.999999999 with a maximum precision of nine decimal places, which isn't the full range of a decimal.

Conversion between this type and the BCL decimal type might be implemented in C# like this:

namespace CustomTypes
{
    public partial class DecimalValue
    {
        private const decimal NanoFactor = 1_000_000_000;
        public DecimalValue(long units, int nanos)
        {
            Units = units;
            Nanos = nanos;
        }

        public static implicit operator decimal(CustomTypes.DecimalValue grpcDecimal)
        {
            return grpcDecimal.Units + grpcDecimal.Nanos / NanoFactor;
        }

        public static implicit operator CustomTypes.DecimalValue(decimal value)
        {
            var units = decimal.ToInt64(value);
            var nanos = decimal.ToInt32((value - units) * NanoFactor);
            return new CustomTypes.DecimalValue(units, nanos);
        }
    }
}

The preceding code:

Adds a partial class for DecimalValue. The partial class is combined with DecimalValue generated from the .proto file. The generated class declares the Units and Nanos properties.
Has implicit operators for converting between DecimalValue and the BCL decimal type.

Collections

Lists

Lists in Protobuf are specified by using the repeated prefix keyword on a field. The following example shows how to create a list:

message Person {
    // ...
    repeated string roles = 8;
}

In the generated code, repeated fields are represented by the Google.Protobuf.Collections.RepeatedField<T> generic type.

public class Person
{
    // ...
    public RepeatedField<string> Roles { get; }
}

RepeatedField<T> implements IList<T>. So you can use LINQ queries or convert it to an array or a list. RepeatedField<T> properties don't have a public setter. Items should be added to the existing collection.

var person = new Person();

// Add one item.
person.Roles.Add("user");

// Add all items from another collection.
var roles = new [] { "admin", "manager" };
person.Roles.Add(roles);

Dictionaries

The .NET IDictionary<TKey,TValue> type is represented in Protobuf using map<key_type, value_type>.

message Person {
    // ...
    map<string, string> attributes = 9;
}

In generated .NET code, map fields are represented by the Google.Protobuf.Collections.MapField<TKey, TValue> generic type. MapField<TKey, TValue> implements IDictionary<TKey,TValue>. Like repeated properties, map properties don't have a public setter. Items should be added to the existing collection.

var person = new Person();

// Add one item.
person.Attributes["created_by"] = "James";

// Add all items from another collection.
var attributes = new Dictionary<string, string>
{
    ["last_modified"] = DateTime.UtcNow.ToString()
};
person.Attributes.Add(attributes);

Unstructured and conditional messages

Protobuf is a contract-first messaging format. An app's messages, including its fields and types, must be specified in .proto files when the app is built. Protobuf's contract-first design is great at enforcing message content but can limit scenarios where a strict contract isn't required:

Messages with unknown payloads. For example, a message with a field that could contain any message.
Conditional messages. For example, a message returned from a gRPC service might be a success result or an error result.
Dynamic values. For example, a message with a field that contains an unstructured collection of values, similar to JSON.

Protobuf offers language features and types to support these scenarios.

Any

The Any type lets you use messages as embedded types without having their .proto definition. To use the Any type, import any.proto.

import "google/protobuf/any.proto";

message Status {
    string message = 1;
    google.protobuf.Any detail = 2;
}

// Create a status with a Person message set to detail.
var status = new ErrorStatus();
status.Detail = Any.Pack(new Person { FirstName = "James" });

// Read Person message from detail.
if (status.Detail.Is(Person.Descriptor))
{
    var person = status.Detail.Unpack<Person>();
    // ...
}

Oneof

oneof fields are a language feature. The compiler handles the oneof keyword when it generates the message class. Using oneof to specify a response message that could either return a Person or Error might look like this:

message Person {
    // ...
}

message Error {
    // ...
}

message ResponseMessage {
  oneof result {
    Error error = 1;
    Person person = 2;
  }
}

Fields within the oneof set must have unique field numbers in the overall message declaration.

When using oneof, the generated C# code includes an enum that specifies which of the fields has been set. You can test the enum to find which field is set. Fields that aren't set return null or the default value, rather than throwing an exception.

var response = await client.GetPersonAsync(new RequestMessage());

switch (response.ResultCase)
{
    case ResponseMessage.ResultOneofCase.Person:
        HandlePerson(response.Person);
        break;
    case ResponseMessage.ResultOneofCase.Error:
        HandleError(response.Error);
        break;
    default:
        throw new ArgumentException("Unexpected result.");
}

Value

The Value type represents a dynamically typed value. It can be either null, a number, a string, a boolean, a dictionary of values (Struct), or a list of values (ValueList). Value is a Protobuf Well-Known Type that uses the previously discussed oneof feature. To use the Value type, import struct.proto.

import "google/protobuf/struct.proto";

message Status {
    // ...
    google.protobuf.Value data = 3;
}

// Create dynamic values.
var status = new Status();
status.Data = Value.ForStruct(new Struct
{
    Fields =
    {
        ["enabled"] = Value.ForBool(true),
        ["metadata"] = Value.ForList(
            Value.ForString("value1"),
            Value.ForString("value2"))
    }
});

// Read dynamic values.
switch (status.Data.KindCase)
{
    case Value.KindOneofCase.StructValue:
        foreach (var field in status.Data.StructValue.Fields)
        {
            // Read struct fields...
        }
        break;
    // ...
}

Using Value directly can be verbose. An alternative way to use Value is with Protobuf's built-in support for mapping messages to JSON. Protobuf's JsonFormatter and JsonWriter types can be used with any Protobuf message. Value is particularly well suited to being converted to and from JSON.

This is the JSON equivalent of the previous code:

// Create dynamic values from JSON.
var status = new Status();
status.Data = Value.Parser.ParseJson(@"{
    ""enabled"": true,
    ""metadata"": [ ""value1"", ""value2"" ]
}");

// Convert dynamic values to JSON.
// JSON can be read with a library like System.Text.Json or Newtonsoft.Json
var json = JsonFormatter.Default.Format(status.Data);
var document = JsonDocument.Parse(json);

Share via

Create Protobuf messages for .NET apps

Protobuf messages

Scalar Value Types

Dates and times

Nullable types

Bytes

Decimals

Creating a custom decimal type for Protobuf

Collections

Lists

Dictionaries

Unstructured and conditional messages

Any

Oneof

Value

Additional resources

Additional resources