Edit

Enable Azure Monitor OpenTelemetry for .NET, Node.js, Python and Java applications

  • Article
  • 36 minutes to read

This article describes how to enable and configure OpenTelemetry-based data collection to power the experiences within Azure Monitor Application Insights. To learn more about OpenTelemetry concepts, see the OpenTelemetry overview or OpenTelemetry FAQ.

OpenTelemetry Release Status

OpenTelemetry offerings are available for .NET, Node.js, Python and Java applications.

Language Release Status
Java ✅ 1
.NET ⚠️ 2
Node.js ⚠️ 2
Python ⚠️ 2

Footnotes

Note

For a feature-by-feature release status, see the FAQ.

Get started

Follow the steps in this section to instrument your application with OpenTelemetry.

Prerequisites

  • Application using an officially supported version of .NET Core or .NET Framework that's at least .NET Framework 4.6.2

Install the client libraries

Install the latest Azure.Monitor.OpenTelemetry.Exporter NuGet package:

dotnet add package --prerelease Azure.Monitor.OpenTelemetry.Exporter

If you get an error like "There are no versions available for the package Azure.Monitor.OpenTelemetry.Exporter," it's probably because the setting of NuGet package sources is missing. Try to specify the source with the -s option:

# Install the latest package with the NuGet package source specified.
dotnet add package --prerelease Azure.Monitor.OpenTelemetry.Exporter -s https://api.nuget.org/v3/index.json

Enable Azure Monitor Application Insights

This section provides guidance that shows how to enable OpenTelemetry.

Instrument with OpenTelemetry

The following code demonstrates how to enable OpenTelemetry in a C# console application by setting up OpenTelemetry TracerProvider. This code must be in the application startup. For ASP.NET Core, it's done typically in the ConfigureServices method of the application Startup class. For ASP.NET applications, it's done typically in Global.asax.cs.

using System.Diagnostics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Trace;

public class Program
{
    private static readonly ActivitySource MyActivitySource = new ActivitySource(
        "OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .AddSource("OTel.AzureMonitor.Demo")
            .AddAzureMonitorTraceExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        using (var activity = MyActivitySource.StartActivity("TestActivity"))
        {
            activity?.SetTag("CustomTag1", "Value1");
            activity?.SetTag("CustomTag2", "Value2");
        }

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Note

The Activity and ActivitySource classes from the System.Diagnostics namespace represent the OpenTelemetry concepts of Span and Tracer, respectively. You create ActivitySource directly by using its constructor instead of by using TracerProvider. Each ActivitySource class must be explicitly connected to TracerProvider by using AddSource(). That's because parts of the OpenTelemetry tracing API are incorporated directly into the .NET runtime. To learn more, see Introduction to OpenTelemetry .NET Tracing API.

Tip

For .NET, Node.js, and Python, you'll need to manually add instrumentation libraries to autocollect telemetry across popular frameworks and libraries. For Java, these instrumentation libraries are already included and no additional steps are required.

Set the Application Insights connection string

You can find your connection string in the Overview Pane of your Application Insights Resource.

Screenshot of the Application Insights connection string.

Here's how you set the connection string.

Replace the <Your Connection String> in the preceding code with the connection string from your Application Insights resource.

Confirm data is flowing

Run your application and open your Application Insights Resource tab in the Azure portal. It might take a few minutes for data to show up in the portal.

Note

If you can't run the application or you aren't getting data as expected, see Troubleshooting.

Screenshot of the Application Insights Overview tab with server requests and server response time highlighted.

Important

If you have two or more services that emit telemetry to the same Application Insights resource, you're required to set Cloud Role Names to represent them properly on the Application Map.

As part of using Application Insights instrumentation, we collect and send diagnostic data to Microsoft. This data helps us run and improve Application Insights. To learn more, see Statsbeat in Azure Application Insights.

Set the Cloud Role Name and the Cloud Role Instance

You might want to update the Cloud Role Name and the Cloud Role Instance from the default values to something that makes sense to your team. They'll appear on the Application Map as the name underneath a node.

Set the Cloud Role Name and the Cloud Role Instance via Resource attributes. Cloud Role Name uses service.namespace and service.name attributes, although it falls back to service.name if service.namespace isn't set. Cloud Role Instance uses the service.instance.id attribute value. For information on standard attributes for resources, see Resource Semantic Conventions.

// Setting role name and role instance
var resourceAttributes = new Dictionary<string, object> {
    { "service.name", "my-service" },
    { "service.namespace", "my-namespace" },
    { "service.instance.id", "my-instance" }};
var resourceBuilder = ResourceBuilder.CreateDefault().AddAttributes(resourceAttributes);
// Done setting role name and role instance

// Set ResourceBuilder on the provider.
var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .SetResourceBuilder(resourceBuilder)
    .AddSource("OTel.AzureMonitor.Demo")
    .AddAzureMonitorTraceExporter(o =>
    {
        o.ConnectionString = "<Your Connection String>";
    })
    .Build();

Enable Sampling

You may want to enable sampling to reduce your data ingestion volume, which reduces your cost. Azure Monitor provides a custom fixed-rate sampler that populates events with a "sampling ratio", which Application Insights converts to "ItemCount". The fixed-rate sampler ensures accurate experiences and event counts. The sampler is designed to preserve your traces across services, and it's interoperable with older Application Insights SDKs. For more information, see Learn More about sampling.

Note

Metrics are unaffected by sampling.

The sampler expects a sample rate of between 0 and 1 inclusive. A rate of 0.1 means approximately 10% of your traces will be sent.

In this example, we utilize the ApplicationInsightsSampler, which offers compatibility with Application Insights SDKs.

dotnet add package --prerelease OpenTelemetry.Extensions.AzureMonitor

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource("OTel.AzureMonitor.Demo")
    .SetSampler(new ApplicationInsightsSampler(0.1F))
    .AddAzureMonitorTraceExporter(o =>
    {
     o.ConnectionString = "<Your Connection String>";
    })
    .Build();

Tip

When using fixed-rate/percentage sampling and you aren't sure what to set the sampling rate as, start at 5% (i.e., 0.05 sampling ratio) and adjust the rate based on the accuracy of the operations shown in the failures and performance blades. A higher rate generally results in higher accuracy. However, ANY sampling will affect accuracy so we recommend alerting on OpenTelemetry metrics, which are unaffected by sampling.

Instrumentation libraries

The following libraries are validated to work with the current release.

Warning

Instrumentation libraries are based on experimental OpenTelemetry specifications, which impacts languages in preview status. Microsoft's preview support commitment is to ensure that the following libraries emit data to Azure Monitor Application Insights, but it's possible that breaking changes or experimental mapping will block some data elements.

Distributed Tracing

Requests

Dependencies

Metrics

Tip

The OpenTelemetry-based offerings currently emit all metrics as Custom Metrics and Performance Counters in Metrics Explorer. For .NET, Node.js, and Python, whatever you set as the meter name becomes the metrics namespace.

Logs

Coming soon.

Footnotes

  • 1: Supports automatic reporting of unhandled exceptions
  • 2: By default, logging is only collected when that logging is performed at the INFO level or higher. To change this level, see the configuration options.
  • 3: By default, logging is only collected when that logging is performed at the WARNING level or higher. To change this level, see the configuration options and specify logging_level.

Collect custom telemetry

This section explains how to collect custom telemetry from your application.

Depending on your language and signal type, there are different ways to collect custom telemetry, including:

  • OpenTelemetry API
  • Language-specific logging/metrics libraries
  • Application Insights Classic API

The following table represents the currently supported custom telemetry types:

Custom Telemetry Types Custom Events Custom Metrics Dependencies Exceptions Page Views Requests Traces
.NET
   OpenTelemetry API Yes Yes Yes
   iLogger API Yes
   AI Classic API
Java
   OpenTelemetry API Yes Yes Yes Yes
   Logback, Log4j, JUL Yes Yes
   Micrometer Metrics Yes
   AI Classic API Yes Yes Yes Yes Yes Yes Yes
Node.js
   OpenTelemetry API Yes Yes Yes Yes
   Winston, Pino, Bunyan Yes
   AI Classic API Yes Yes Yes Yes Yes Yes Yes
Python
   OpenTelemetry API Yes Yes Yes Yes
   Python Logging Module Yes

Note

Application Insights Java 3.x listens for telemetry that's sent to the Application Insights Classic API. Similarly, Application Insights Node.js 3.x collects events created with the Application Insights Classic API. This makes upgrading easier and fills a gap in our custom telemetry support until all custom telemetry types are supported via the OpenTelemetry API.

Add Custom Metrics

Note

Custom Metrics are under preview in Azure Monitor Application Insights. Custom metrics without dimensions are available by default. To view and alert on dimensions, you need to opt-in.

You may want to collect metrics beyond what is collected by instrumentation libraries.

The OpenTelemetry API offers six metric "instruments" to cover various metric scenarios and you'll need to pick the correct "Aggregation Type" when visualizing metrics in Metrics Explorer. This requirement is true when using the OpenTelemetry Metric API to send metrics and when using an instrumentation library.

The following table shows the recommended aggregation types for each of the OpenTelemetry Metric Instruments.

OpenTelemetry Instrument Azure Monitor Aggregation Type
Counter Sum
Asynchronous Counter Sum
Histogram Min, Max, Average, Sum and Count
Asynchronous Gauge Average
UpDownCounter Sum
Asynchronous UpDownCounter Sum

Caution

Aggregation types beyond what's shown in the table typically aren't meaningful.

The OpenTelemetry Specification describes the instruments and provides examples of when you might use each one.

Tip

The histogram is the most versatile and most closely equivalent to the Application Insights Track Metric Classic API. Azure Monitor currently flattens the histogram instrument into our five supported aggregation types, and support for percentiles is underway. Although less versatile, other OpenTelemetry instruments have a lesser impact on your application's performance.

Histogram Example

using System.Diagnostics.Metrics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Metrics;

public class Program
{
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        Histogram<long> myFruitSalePrice = meter.CreateHistogram<long>("FruitSalePrice");

        var rand = new Random();
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "green"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Counter Example

using System.Diagnostics.Metrics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Metrics;

public class Program
{
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        Counter<long> myFruitCounter = meter.CreateCounter<long>("MyFruitCounter");

        myFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
        myFruitCounter.Add(2, new("name", "lemon"), new("color", "yellow"));
        myFruitCounter.Add(1, new("name", "lemon"), new("color", "yellow"));
        myFruitCounter.Add(2, new("name", "apple"), new("color", "green"));
        myFruitCounter.Add(5, new("name", "apple"), new("color", "red"));
        myFruitCounter.Add(4, new("name", "lemon"), new("color", "yellow"));

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Gauge Example

using System.Diagnostics.Metrics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Metrics;

public class Program
{
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter(o =>
            {
                o.ConnectionString = "<Your Connection String>";
            })
            .Build();

        var process = Process.GetCurrentProcess();
        
        ObservableGauge<int> myObservableGauge = meter.CreateObservableGauge("Thread.State", () => GetThreadState(process));

        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
    
    private static IEnumerable<Measurement<int>> GetThreadState(Process process)
    {
        foreach (ProcessThread thread in process.Threads)
        {
            yield return new((int)thread.ThreadState, new("ProcessId", process.Id), new("ThreadId", thread.Id));
        }
    }
}

Add Custom Exceptions

Select instrumentation libraries automatically report exceptions to Application Insights. However, you may want to manually report exceptions beyond what instrumentation libraries report. For instance, exceptions caught by your code aren't ordinarily reported. You may wish to report them to draw attention in relevant experiences including the failures section and end-to-end transaction views.

using (var activity = activitySource.StartActivity("ExceptionExample"))
{
    try
    {
        throw new Exception("Test exception");
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error);
        activity?.RecordException(ex);
    }
}

Add Custom Spans

You may want to add a custom span when there's a dependency request that's not already collected by an instrumentation library or an application process that you wish to model as a span on the end-to-end transaction view.

Coming soon.

Send custom telemetry using the Application Insights Classic API

We recommend you use the OpenTelemetry APIs whenever possible, but there may be some scenarios when you have to use the Application Insights Classic APIs.

This is not available in .NET.

Modify telemetry

This section explains how to modify telemetry.

Add span attributes

These attributes might include adding a custom property to your telemetry. You might also use attributes to set optional fields in the Application Insights schema, like Client IP.

Add a custom property to a Span

Any attributes you add to spans are exported as custom properties. They populate the customDimensions field in the requests, dependencies, traces, or exceptions table.

To add span attributes, use either of the following two ways:

Tip

The advantage of using options provided by instrumentation libraries, when they're available, is that the entire context is available. As a result, users can select to add or filter more attributes. For example, the enrich option in the HttpClient instrumentation library gives users access to the httpRequestMessage itself. They can select anything from it and store it as an attribute.

  1. Many instrumentation libraries provide an enrich option. For guidance, see the readme files of individual instrumentation libraries:

  2. Use a custom processor:

Tip

Add the processor shown here before the Azure Monitor Exporter.

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
        .AddSource("OTel.AzureMonitor.Demo")
        .AddProcessor(new ActivityEnrichingProcessor())
        .AddAzureMonitorTraceExporter(o =>
        {
                o.ConnectionString = "<Your Connection String>"
        })
        .Build();

Add ActivityEnrichingProcessor.cs to your project with the following code:

using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;

public class ActivityEnrichingProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        // The updated activity will be available to all processors which are called after this processor.
        activity.DisplayName = "Updated-" + activity.DisplayName;
        activity.SetTag("CustomDimension1", "Value1");
        activity.SetTag("CustomDimension2", "Value2");
    }
}

Set the user IP

You can populate the client_IP field for requests by setting the http.client_ip attribute on the span. Application Insights uses the IP address to generate user location attributes and then discards it by default.

Use the add custom property example, but replace the following lines of code in ActivityEnrichingProcessor.cs:

// only applicable in case of activity.Kind == Server
activity.SetTag("http.client_ip", "<IP Address>");

Set the user ID or authenticated user ID

You can populate the user_Id or user_AuthenticatedId field for requests by using the guidance below. User ID is an anonymous user identifier. Authenticated User ID is a known user identifier.

Important

Consult applicable privacy laws before you set the Authenticated User ID.

Coming soon.

Add Log Attributes

Coming soon.

Filter telemetry

You might use the following ways to filter out telemetry before it leaves your application.

  1. Many instrumentation libraries provide a filter option. For guidance, see the readme files of individual instrumentation libraries:

  2. Use a custom processor:

    using var tracerProvider = Sdk.CreateTracerProviderBuilder()
        .AddSource("OTel.AzureMonitor.Demo")
        .AddProcessor(new ActivityFilteringProcessor())
        .AddAzureMonitorTraceExporter(o =>
        {
                o.ConnectionString = "<Your Connection String>"
        })
        .Build();

    Add ActivityFilteringProcessor.cs to your project with the following code:

    using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;

public class ActivityFilteringProcessor : BaseProcessor<Activity>
{
    public override void OnStart(Activity activity)
    {
        // prevents all exporters from exporting internal activities
        if (activity.Kind == ActivityKind.Internal)
        {
            activity.IsAllDataRequested = false;
        }
    }
}

  3. If a particular source isn't explicitly added by using AddSource("ActivitySourceName"), then none of the activities created by using that source will be exported.

Get the trace ID or span ID

You might want to get the trace ID or span ID. If you have logs that are sent to a different destination besides Application Insights, you might want to add the trace ID or span ID to enable better correlation when you debug and diagnose issues.

Coming soon.

Enable the OTLP Exporter

You might want to enable the OpenTelemetry Protocol (OTLP) Exporter alongside your Azure Monitor Exporter to send your telemetry to two locations.

Note

The OTLP Exporter is shown for convenience only. We don't officially support the OTLP Exporter or any components or third-party experiences downstream of it.

  1. Install the OpenTelemetry.Exporter.OpenTelemetryProtocol package along with Azure.Monitor.OpenTelemetry.Exporter in your project.

  2. Add the following code snippet. This example assumes you have an OpenTelemetry Collector with an OTLP receiver running. For details, see the example on GitHub.

    // Sends data to Application Insights as well as OTLP
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
        .AddSource("OTel.AzureMonitor.Demo")
        .AddAzureMonitorTraceExporter(o =>
        {
            o.ConnectionString = "<Your Connection String>"
        })
        .AddOtlpExporter()
        .Build();

Configuration

Offline Storage and Automatic Retries

To improve reliability and resiliency, Azure Monitor OpenTelemetry-based offerings write to offline/local storage by default when an application loses its connection with Application Insights. It saves the application telemetry to disk and periodically tries to send it again for up to 48 hours. In addition to exceeding the allowable time, telemetry will occasionally be dropped in high-load applications when the maximum file size is exceeded or the SDK doesn't have an opportunity to clear out the file. If we need to choose, the product will save more recent events over old ones. Learn More

By default, the AzureMonitorExporter uses one of the following locations for offline storage (listed in order of precedence):

  • Windows
    • %LOCALAPPDATA%\Microsoft\AzureMonitor
    • %TEMP%\Microsoft\AzureMonitor
  • Non-Windows
    • %TMPDIR%/Microsoft/AzureMonitor
    • /var/tmp/Microsoft/AzureMonitor
    • /tmp/Microsoft/AzureMonitor

To override the default directory, you should set AzureMonitorExporterOptions.StorageDirectory.

For example:

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddAzureMonitorTraceExporter(o => {
        o.ConnectionString = "<Your Connection String>";
        o.StorageDirectory = "C:\\SomeDirectory";
    })
    .Build();

To disable this feature, you should set AzureMonitorExporterOptions.DisableOfflineStorage = true.

Troubleshooting

This section provides help with troubleshooting.

Enable diagnostic logging

The Azure Monitor Exporter uses EventSource for its own internal logging. The exporter logs are available to any EventListener by opting into the source named OpenTelemetry-AzureMonitor-Exporter. For troubleshooting steps, see OpenTelemetry Troubleshooting.

Known issues

Known issues for the Azure Monitor OpenTelemetry Exporters include:

  • Operation name is missing on dependency telemetry, which adversely affects failures and performance tab experience.
  • Device model is missing on request and dependency telemetry, which adversely affects device cohort analysis.
  • Database server name is left out of dependency name, which incorrectly aggregates tables with the same name on different servers.

Test connectivity between your application host and the ingestion service

Application Insights SDKs and agents send telemetry to get ingested as REST calls to our ingestion endpoints. You can test connectivity from your web server or application host machine to the ingestion service endpoints by using raw REST clients from PowerShell or curl commands. See Troubleshoot missing application telemetry in Azure Monitor Application Insights.

Support

To get support:

  • Review troubleshooting steps in this article.
  • For Azure support issues, open an Azure support ticket.

OpenTelemetry feedback

To provide feedback:

Next steps