Edit

Enable Azure Monitor OpenTelemetry for .NET, Node.js, and Python applications (preview)

  • Article
  • 34 minutes to read

The Azure Monitor OpenTelemetry Exporter is a component that sends traces, and metrics (and eventually all application telemetry) to Azure Monitor Application Insights. To learn more about OpenTelemetry concepts, see the OpenTelemetry overview or OpenTelemetry FAQ.

This article describes how to enable and configure the OpenTelemetry-based Azure Monitor Preview offerings. After you finish the instructions in this article, you'll be able to send OpenTelemetry traces and metrics to Azure Monitor Application Insights.

Important

The Azure Monitor OpenTelemetry-based Offerings for .NET, Node.js, and Python applications are currently in preview. See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Limitations of the preview release

Consider whether this preview is right for you. It enables distributed tracing, metrics and excludes:

If you require a full-feature experience, use the existing Application Insights ASP.NET, or ASP.NET Core SDK until the OpenTelemetry-based offering matures.

Get started

Follow the steps in this section to instrument your application with OpenTelemetry.

Prerequisites

  • Application using an officially supported version of .NET Core or .NET Framework that's at least .NET Framework 4.6.2

Install the client libraries

Install the latest Azure.Monitor.OpenTelemetry.Exporter NuGet package:

dotnet add package --prerelease Azure.Monitor.OpenTelemetry.Exporter

If you get an error like "There are no versions available for the package Azure.Monitor.OpenTelemetry.Exporter," it's probably because the setting of NuGet package sources is missing. Try to specify the source with the -s option:

# Install the latest package with the NuGet package source specified.
dotnet add package --prerelease Azure.Monitor.OpenTelemetry.Exporter -s https://api.nuget.org/v3/index.json

Enable Azure Monitor Application Insights

This section provides guidance that shows how to enable OpenTelemetry.

Add OpenTelemetry instrumentation code

The following code demonstrates how to enable OpenTelemetry in a C# console application by setting up OpenTelemetry TracerProvider. This code must be in the application startup. For ASP.NET Core, it's done typically in the ConfigureServices method of the application Startup class. For ASP.NET applications, it's done typically in Global.asax.cs.

using System.Diagnostics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Trace;

public class Program
{
    private static readonly ActivitySource MyActivitySource = new ActivitySource(
        "OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .AddSource("OTel.AzureMonitor.Demo")
            .AddAzureMonitorTraceExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        using (var activity = MyActivitySource.StartActivity("TestActivity"))
        {
            activity?.SetTag("CustomTag1", "Value1");
            activity?.SetTag("CustomTag2", "Value2");
        }

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Note

The Activity and ActivitySource classes from the System.Diagnostics namespace represent the OpenTelemetry concepts of Span and Tracer, respectively. You create ActivitySource directly by using its constructor instead of by using TracerProvider. Each ActivitySource class must be explicitly connected to TracerProvider by using AddSource(). That's because parts of the OpenTelemetry tracing API are incorporated directly into the .NET runtime. To learn more, see Introduction to OpenTelemetry .NET Tracing API.

Tip

Add instrumentation libraries to autocollect telemetry across popular frameworks and libraries.

Set the Application Insights connection string

Replace the placeholder <Your Connection String> in the preceding code with the connection string from your Application Insights resource.

Screenshot of the Application Insights connection string.

Confirm data is flowing

Run your application and open your Application Insights Resource tab in the Azure portal. It might take a few minutes for data to show up in the portal.

Note

If you can't run the application or you aren't getting data as expected, see Troubleshooting.

Screenshot of the Application Insights Overview tab with server requests and server response time highlighted.

Important

If you have two or more services that emit telemetry to the same Application Insights resource, you're required to set Cloud Role Names to represent them properly on the Application Map.

As part of using Application Insights instrumentation, we collect and send diagnostic data to Microsoft. This data helps us run and improve Application Insights. You may disable nonessential data collection. To learn more, see Statsbeat in Azure Application Insights.

Set the Cloud Role Name and the Cloud Role Instance

You might set the Cloud Role Name and the Cloud Role Instance via Resource attributes. This step updates Cloud Role Name and Cloud Role Instance from their default values to something that makes sense to your team. They'll appear on the Application Map as the name underneath a node. Cloud Role Name uses service.namespace and service.name attributes, although it falls back to service.name if service.namespace isn't set. Cloud Role Instance uses the service.instance.id attribute value.

// Setting role name and role instance
var resourceAttributes = new Dictionary<string, object> {
    { "service.name", "my-service" },
    { "service.namespace", "my-namespace" },
    { "service.instance.id", "my-instance" }};
var resourceBuilder = ResourceBuilder.CreateDefault().AddAttributes(resourceAttributes);
// Done setting role name and role instance

// Set ResourceBuilder on the provider.
var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .SetResourceBuilder(resourceBuilder)
    .AddSource("OTel.AzureMonitor.Demo")
    .AddAzureMonitorTraceExporter(o =>
    {
        o.ConnectionString = "<Your Connection String>";
    })
    .Build();

For information on standard attributes for resources, see Resource Semantic Conventions.

Enable Sampling

You may want to enable sampling to reduce your data ingestion volume, which reduces your cost. Azure Monitor provides a custom fixed-rate sampler that populates events with a "sampling ratio", which Application Insights converts to "ItemCount". The fixed-rate sampler ensures accurate experiences and event counts. The sampler is designed to preserve your traces across services, and it's interoperable with older Application Insights SDKs. The sampler expects a sample rate of between 0 and 1 inclusive. A rate of 0.1 means approximately 10% of your traces will be sent. For more information, see Learn More about sampling.

Note

Metrics are unaffected by sampling.

In this example, we utilize the ApplicationInsightsSampler, which offers compatibility with Application Insights SDKs.

dotnet add package --prerelease OpenTelemetry.Extensions.AzureMonitor

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource("OTel.AzureMonitor.Demo")
    .SetSampler(new ApplicationInsightsSampler(0.1F))
    .AddAzureMonitorTraceExporter(o =>
    {
     o.ConnectionString = "<Your Connection String>";
    })
    .Build();

Tip

If you're not sure where to set the sampling rate, start at 5% (i.e., 0.05 sampling ratio) and adjust the rate based on the accuracy of the operations shown in the failures and performance panes. A higher rate generally results in higher accuracy. However, ANY sampling will affect accuracy so we recommend alerting on OpenTelemetry metrics, which are unaffected by sampling.

Instrumentation libraries

The following libraries are validated to work with the preview release.

Warning

Instrumentation libraries are based on experimental OpenTelemetry specifications. Microsoft's preview support commitment is to ensure that the following libraries emit data to Azure Monitor Application Insights, but it's possible that breaking changes or experimental mapping will block some data elements.

Distributed Tracing

Requests

Dependencies

(1) Supports automatic reporting (as SpanEvent) of unhandled exceptions

Metrics

Tip

The OpenTelemetry-based offerings currently emit all metrics as Custom Metrics in Metrics Explorer. Whatever you set as the meter name becomes the metrics namespace.

Modify telemetry

This section explains how to modify telemetry.

Add span attributes

To add span attributes, use either of the following two ways:

These attributes might include adding a custom property to your telemetry. You might also use attributes to set optional fields in the Application Insights schema, like Client IP.

Tip

The advantage of using options provided by instrumentation libraries, when they're available, is that the entire context is available. As a result, users can select to add or filter more attributes. For example, the enrich option in the HttpClient instrumentation library gives users access to the httpRequestMessage itself. They can select anything from it and store it as an attribute.

Add a custom property to a Trace

Any attributes you add to spans are exported as custom properties. They populate the customDimensions field in the requests or the dependencies tables in Application Insights.

  1. Many instrumentation libraries provide an enrich option. For guidance, see the readme files of individual instrumentation libraries:

  2. Use a custom processor:

Tip

Add the processor shown here before the Azure Monitor Exporter.

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
        .AddSource("OTel.AzureMonitor.Demo")
        .AddProcessor(new ActivityEnrichingProcessor())
        .AddAzureMonitorTraceExporter(o =>
        {
                o.ConnectionString = "<Your Connection String>"
        })
        .Build();

Add ActivityEnrichingProcessor.cs to your project with the following code:

using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;

public class ActivityEnrichingProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        // The updated activity will be available to all processors which are called after this processor.
        activity.DisplayName = "Updated-" + activity.DisplayName;
        activity.SetTag("CustomDimension1", "Value1");
        activity.SetTag("CustomDimension2", "Value2");
    }
}

Set the user IP

You can populate the client_IP field for requests by setting the http.client_ip attribute on the span. Application Insights uses the IP address to generate user location attributes and then discards it by default.

Use the add custom property example, but replace the following lines of code in ActivityEnrichingProcessor.cs:

// only applicable in case of activity.Kind == Server
activity.SetTag("http.client_ip", "<IP Address>");

Filter telemetry

You might use the following ways to filter out telemetry before it leaves your application.

  1. Many instrumentation libraries provide a filter option. For guidance, see the readme files of individual instrumentation libraries:

  2. Use a custom processor:

    using var tracerProvider = Sdk.CreateTracerProviderBuilder()
        .AddSource("OTel.AzureMonitor.Demo")
        .AddProcessor(new ActivityFilteringProcessor())
        .AddAzureMonitorTraceExporter(o =>
        {
                o.ConnectionString = "<Your Connection String>"
        })
        .Build();

    Add ActivityFilteringProcessor.cs to your project with the following code:

    using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;

public class ActivityFilteringProcessor : BaseProcessor<Activity>
{
    public override void OnStart(Activity activity)
    {
        // prevents all exporters from exporting internal activities
        if (activity.Kind == ActivityKind.Internal)
        {
            activity.IsAllDataRequested = false;
        }
    }
}

  3. If a particular source isn't explicitly added by using AddSource("ActivitySourceName"), then none of the activities created by using that source will be exported.

Custom telemetry

This section explains how to collect custom telemetry from your application.

Add Custom Metrics

Note

Custom Metrics are under preview in Azure Monitor Application Insights. Custom metrics without dimensions are available by default. To view and alert on dimensions, you need to opt-in.

You may want to collect metrics beyond what is collected by instrumentation libraries.

The OpenTelemetry API offers six metric "instruments" to cover various metric scenarios and you'll need to pick the correct "Aggregation Type" when visualizing metrics in Metrics Explorer. This requirement is true when using the OpenTelemetry Metric API to send metrics and when using an instrumentation library.

The following table shows the recommended aggregation types for each of the OpenTelemetry Metric Instruments.

OpenTelemetry Instrument Azure Monitor Aggregation Type
Counter Sum
Asynchronous Counter Sum
Histogram Min, Max, Average, Sum and Count
Asynchronous Gauge Average
UpDownCounter Sum
Asynchronous UpDownCounter Sum

Caution

Aggregation types beyond what's shown in the table typically aren't meaningful.

The OpenTelemetry Specification describes the instruments and provides examples of when you might use each one.

Tip

The histogram is the most versatile and most closely equivalent to the prior Application Insights Track Metric API. Azure Monitor currently flattens the histogram instrument into our five supported aggregation types, and support for percentiles is underway. Although less versatile, other OpenTelemetry instruments have a lesser impact on your application's performance.

Histogram Example

using System.Diagnostics.Metrics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Metrics;

public class Program
{
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        Histogram<long> myFruitSalePrice = meter.CreateHistogram<long>("FruitSalePrice");

        var rand = new Random();
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "green"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Counter Example

using System.Diagnostics.Metrics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Metrics;

public class Program
{
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        Counter<long> myFruitCounter = meter.CreateCounter<long>("MyFruitCounter");

        myFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
        myFruitCounter.Add(2, new("name", "lemon"), new("color", "yellow"));
        myFruitCounter.Add(1, new("name", "lemon"), new("color", "yellow"));
        myFruitCounter.Add(2, new("name", "apple"), new("color", "green"));
        myFruitCounter.Add(5, new("name", "apple"), new("color", "red"));
        myFruitCounter.Add(4, new("name", "lemon"), new("color", "yellow"));

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Gauge Example

using System.Diagnostics.Metrics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Metrics;

public class Program
{
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        var process = Process.GetCurrentProcess();
        
        ObservableGauge<int> myObservableGauge = meter.CreateObservableGauge("Thread.State", () => GetThreadState(process));

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
    
    private static IEnumerable<Measurement<int>> GetThreadState(Process process)
    {
        foreach (ProcessThread thread in process.Threads)
        {
            yield return new((int)thread.ThreadState, new("ProcessId", process.Id), new("ThreadId", thread.Id));
        }
    }
}

Add Custom Exceptions

Select instrumentation libraries automatically support exceptions to Application Insights. However, you may want to manually report exceptions beyond what instrumentation libraries report. For instance, exceptions caught by your code aren't* ordinarily reported. You may wish to report them to draw attention in relevant experiences including the failures section and end-to-end transaction views.

using (var activity = activitySource.StartActivity("ExceptionExample"))
{
    try
    {
        throw new Exception("Test exception");
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error);
        activity?.RecordException(ex);
    }
}

Enable the OTLP Exporter

You might want to enable the OpenTelemetry Protocol (OTLP) Exporter alongside your Azure Monitor Exporter to send your telemetry to two locations.

Note

The OTLP Exporter is shown for convenience only. We don't officially support the OTLP Exporter or any components or third-party experiences downstream of it. We suggest you open an issue with the OpenTelemetry-Collector for OpenTelemetry issues outside the Azure support boundary.

  1. Install the OpenTelemetry.Exporter.OpenTelemetryProtocol package along with Azure.Monitor.OpenTelemetry.Exporter in your project.

  2. Add the following code snippet. This example assumes you have an OpenTelemetry Collector with an OTLP receiver running. For details, see the example on GitHub.

    // Sends data to Application Insights as well as OTLP
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
        .AddSource("OTel.AzureMonitor.Demo")
        .AddAzureMonitorTraceExporter(o =>
        {
            o.ConnectionString = "<Your Connection String>"
        })
        .AddOtlpExporter()
        .Build();

Configuration

Offline Storage and Automatic Retries

To improve reliability and resiliency, Azure Monitor OpenTelemetry-based offerings write to offline/local storage by default when an application loses its connection with Application Insights. It saves the application telemetry for 48 hours and periodically tries to send it again. In addition to exceeding the allowable time, telemetry will occasionally be dropped in high-load applications when the maximum file size is exceeded or the SDK doesn't have an opportunity to clear out the file. If we need to choose, the product will save more recent events over old ones. In some cases, you may wish to disable this feature to optimize application performance. Learn More

By default, the AzureMonitorExporter uses one of the following locations for offline storage (listed in order of precedence):

  • Windows
    • %LOCALAPPDATA%\Microsoft\AzureMonitor
    • %TEMP%\Microsoft\AzureMonitor
  • Non-Windows
    • %TMPDIR%/Microsoft/AzureMonitor
    • /var/tmp/Microsoft/AzureMonitor
    • /tmp/Microsoft/AzureMonitor

To override the default directory, you should set AzureMonitorExporterOptions.StorageDirectory.

For example:

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddAzureMonitorTraceExporter(o => {
        o.ConnectionString = "<Your Connection String>";
        o.StorageDirectory = "C:\\SomeDirectory";
    })
    .Build();

To disable this feature, you should set AzureMonitorExporterOptions.DisableOfflineStorage = true.

Troubleshooting

This section provides help with troubleshooting.

Enable diagnostic logging

The Azure Monitor Exporter uses EventSource for its own internal logging. The exporter logs are available to any EventListener by opting into the source named OpenTelemetry-AzureMonitor-Exporter. For troubleshooting steps, see OpenTelemetry Troubleshooting.

Known issues

Known issues for the Azure Monitor OpenTelemetry Exporters include:

  • Operation name is missing on dependency telemetry, which adversely affects failures and performance tab experience.
  • Device model is missing on request and dependency telemetry, which adversely affects device cohort analysis.
  • Database server name is left out of dependency name, which incorrectly aggregates tables with the same name on different servers.

Test connectivity between your application host and the ingestion service

Application Insights SDKs and agents send telemetry to get ingested as REST calls to our ingestion endpoints. You can test connectivity from your web server or application host machine to the ingestion service endpoints by using raw REST clients from PowerShell or curl commands. See Troubleshoot missing application telemetry in Azure Monitor Application Insights.

Support

To get support:

  • Review troubleshooting steps in this article.
  • For Azure support issues, open an Azure support ticket.

For OpenTelemetry issues, contact the OpenTelemetry .NET community directly.

OpenTelemetry feedback

To provide feedback:

Next steps