Logging and diagnostics in gRPC on .NET

By James Newton-King

This article provides guidance for gathering diagnostics from a gRPC app to help troubleshoot issues. Topics covered include:

• Logging - Structured logs written to .NET Core logging. ILogger is used by app frameworks to write logs, and by users for their own logging in an app.
• Tracing - Events related to an operation written using DiaganosticSource and Activity. Traces from diagnostic source are commonly used to collect app telemetry by libraries such as Application Insights and OpenTelemetry.
• Metrics - Representation of data measures over intervals of time, for example, requests per second. Metrics are emitted using EventCounter and can be observed using dotnet-counters command-line tool or with Application Insights.

Logging

gRPC services and the gRPC client write logs using .NET Core logging. Logs are a good place to start when debugging unexpected behavior in service and client apps.

gRPC services logging

Warning

Server-side logs may contain sensitive information from your app. Never post raw logs from production apps to public forums like GitHub.

Since gRPC services are hosted on ASP.NET Core, it uses the ASP.NET Core logging system. In the default configuration, gRPC logs minimal information, but logging can be configured. See the documentation on ASP.NET Core logging for details on configuring ASP.NET Core logging.

gRPC adds logs under the Grpc category. To enable detailed logs from gRPC, configure the Grpc prefixes to the Debug level in the appsettings.json file by adding the following items to the LogLevel subsection in Logging:

{
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "System": "Information",
      "Microsoft": "Information",
      "Grpc": "Debug"
    }
  }
}

Logging can also be configured in Program.cs with ConfigureLogging:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureLogging(logging =>
        {
            logging.AddFilter("Grpc", LogLevel.Debug);
        })
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });

When not using JSON-based configuration, set the following configuration value in the configuration system:

• Logging:LogLevel:Grpc = Debug

Check the documentation for your configuration system to determine how to specify nested configuration values. For example, when using environment variables, two _ characters are used instead of the : (for example, Logging__LogLevel__Grpc).

We recommend using the Debug level when gathering detailed diagnostics for an app. The Trace level produces low-level diagnostics and is rarely needed to diagnose issues.

Sample logging output

Here is an example of console output at the Debug level of a gRPC service:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST https://localhost:5001/Greet.Greeter/SayHello application/grpc
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /Greet.Greeter/SayHello'
dbug: Grpc.AspNetCore.Server.ServerCallHandler[1]
      Reading message.
info: GrpcService.GreeterService[0]
      Hello World
dbug: Grpc.AspNetCore.Server.ServerCallHandler[6]
      Sending message.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /Greet.Greeter/SayHello'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished in 1.4113ms 200 application/grpc

Access server-side logs

How server-side logs are accessed depends on the app's environment.

As a console app

If you're running in a console app, the Console logger should be enabled by default. gRPC logs will appear in the console.

Other environments

If the app is deployed to another environment (for example, Docker, Kubernetes, or Windows Service), see Logging in .NET Core and ASP.NET Core for more information on how to configure logging providers suitable for the environment.

gRPC client logging

Warning

Client-side logs may contain sensitive information from your app. Never post raw logs from production apps to public forums like GitHub.

To get logs from the .NET client, set the GrpcChannelOptions.LoggerFactory property when the client's channel is created. When calling a gRPC service from an ASP.NET Core app, the logger factory can be resolved from dependency injection (DI):

[ApiController]
[Route("[controller]")]
public class GreetingController : ControllerBase
{
    private ILoggerFactory _loggerFactory;

    public GreetingController(ILoggerFactory loggerFactory)
    {
        _loggerFactory = loggerFactory;
    }

    [HttpGet]
    public async Task<ActionResult<string>> Get(string name)
    {
        var channel = GrpcChannel.ForAddress("https://localhost:5001",
            new GrpcChannelOptions { LoggerFactory = _loggerFactory });
        var client = new Greeter.GreeterClient(channel);

        var reply = await client.SayHelloAsync(new HelloRequest { Name = name });
        return Ok(reply.Message);
    }
}

An alternative way to enable client logging is to use the gRPC client factory to create the client. A gRPC client registered with the client factory and resolved from DI will automatically use the app's configured logging.

If the app isn't using DI, then create a new ILoggerFactory instance with LoggerFactory.Create. To access this method, add the Microsoft.Extensions.Logging package to your app.

var loggerFactory = LoggerFactory.Create(logging =>
{
    logging.AddConsole();
    logging.SetMinimumLevel(LogLevel.Debug);
});

var channel = GrpcChannel.ForAddress("https://localhost:5001",
    new GrpcChannelOptions { LoggerFactory = loggerFactory });

var client = Greeter.GreeterClient(channel);

gRPC client log scopes

The gRPC client adds a logging scope to logs made during a gRPC call. The scope has metadata related to the gRPC call:

• GrpcMethodType - The gRPC method type. Possible values are names from Grpc.Core.MethodType enum. For example, Unary.
• GrpcUri - The relative URI of the gRPC method. For example, /greet.Greeter/SayHellos.

Sample logging output

Here is an example of console output at the Debug level of a gRPC client:

dbug: Grpc.Net.Client.Internal.GrpcCall[1]
      Starting gRPC call. Method type: 'Unary', URI: 'https://localhost:5001/Greet.Greeter/SayHello'.
dbug: Grpc.Net.Client.Internal.GrpcCall[6]
      Sending message.
dbug: Grpc.Net.Client.Internal.GrpcCall[1]
      Reading message.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
      Finished gRPC call.

Tracing

gRPC services and the gRPC client provide information about gRPC calls using DiagnosticSource and Activity.

• .NET gRPC uses an activity to represent a gRPC call.
• Tracing events are written to the diagnostic source at the start and stop of the gRPC call activity.
• Tracing doesn't capture information about when messages are sent over the lifetime of gRPC streaming calls.

gRPC service tracing

gRPC services are hosted on ASP.NET Core, which reports events about incoming HTTP requests. gRPC specific metadata is added to the existing HTTP request diagnostics that ASP.NET Core provides.

• Diagnostic source name is Microsoft.AspNetCore.
• Activity name is Microsoft.AspNetCore.Hosting.HttpRequestIn.
  • Name of the gRPC method invoked by the gRPC call is added as a tag with the name grpc.method.
  • Status code of the gRPC call when it is complete is added as a tag with the name grpc.status_code.

gRPC client tracing

The .NET gRPC client uses HttpClient to make gRPC calls. Although HttpClient writes diagnostic events, the .NET gRPC client provides a custom diagnostic source, activity, and events so that complete information about a gRPC call can be collected.

• Diagnostic source name is Grpc.Net.Client.
• Activity name is Grpc.Net.Client.GrpcOut.
  • Name of the gRPC method invoked by the gRPC call is added as a tag with the name grpc.method.
  • Status code of the gRPC call when it is complete is added as a tag with the name grpc.status_code.

Collecting tracing

The easiest way to use DiagnosticSource is to configure a telemetry library such as Application Insights or OpenTelemetry in your app. The library will process information about gRPC calls along-side other app telemetry.

Tracing can be viewed in a managed service like Application Insights, or run as your own distributed tracing system. OpenTelemetry supports exporting tracing data to Jaeger and Zipkin.

DiagnosticSource can consume tracing events in code using DiagnosticListener. For information about listening to a diagnostic source with code, see the DiagnosticSource user's guide.

Note

Telemetry libraries do not capture gRPC specific Grpc.Net.Client.GrpcOut telemetry currently. Work to improve telemetry libraries capturing this tracing is ongoing.

Metrics

Metrics is a representation of data measures over intervals of time, for example, requests per second. Metrics data allows observation of the state of an app at a high level. .NET gRPC metrics are emitted using EventCounter.

gRPC service metrics

gRPC server metrics are reported on Grpc.AspNetCore.Server event source.

Name	Description
total-calls	Total Calls
current-calls	Current Calls
calls-failed	Total Calls Failed
calls-deadline-exceeded	Total Calls Deadline Exceeded
messages-sent	Total Messages Sent
messages-received	Total Messages Received
calls-unimplemented	Total Calls Unimplemented

ASP.NET Core also provides its own metrics on Microsoft.AspNetCore.Hosting event source.

gRPC client metrics

gRPC client metrics are reported on Grpc.Net.Client event source.

Name	Description
total-calls	Total Calls
current-calls	Current Calls
calls-failed	Total Calls Failed
calls-deadline-exceeded	Total Calls Deadline Exceeded
messages-sent	Total Messages Sent
messages-received	Total Messages Received

Observe metrics

dotnet-counters is a performance monitoring tool for ad-hoc health monitoring and first-level performance investigation. Monitor a .NET app with either Grpc.AspNetCore.Server or Grpc.Net.Client as the provider name.

> dotnet-counters monitor --process-id 1902 Grpc.AspNetCore.Server

Press p to pause, r to resume, q to quit.
    Status: Running
[Grpc.AspNetCore.Server]
    Total Calls                                 300
    Current Calls                               5
    Total Calls Failed                          0
    Total Calls Deadline Exceeded               0
    Total Messages Sent                         295
    Total Messages Received                     300
    Total Calls Unimplemented                   0

Another way to observe gRPC metrics is to capture counter data using Application Insights's Microsoft.ApplicationInsights.EventCounterCollector package. Once setup, Application Insights collects common .NET counters at runtime. gRPC's counters are not collected by default, but App Insights can be customized to include additional counters.

Specify the gRPC counters for Application Insight to collect in Startup.cs:

    using Microsoft.ApplicationInsights.Extensibility.EventCounterCollector;

    public void ConfigureServices(IServiceCollection services)
    {
        //... other code...

        services.ConfigureTelemetryModule<EventCounterCollectionModule>(
            (module, o) =>
            {
                // Configure App Insights to collect gRPC counters gRPC services hosted in an ASP.NET Core app
                module.Counters.Add(new EventCounterCollectionRequest("Grpc.AspNetCore.Server", "current-calls"));
                module.Counters.Add(new EventCounterCollectionRequest("Grpc.AspNetCore.Server", "total-calls"));
                module.Counters.Add(new EventCounterCollectionRequest("Grpc.AspNetCore.Server", "calls-failed"));
            }
        );
    }

Additional resources

• Logging in .NET Core and ASP.NET Core
• gRPC for .NET configuration
• gRPC client factory integration in .NET