Add, modify, and filter OpenTelemetry

This article provides guidance on how to add, modify, and filter OpenTelemetry for applications using Azure Monitor Application Insights.

To learn more about OpenTelemetry concepts, see the OpenTelemetry overview or OpenTelemetry FAQ.

Automatic data collection

The distros automatically collect data by bundling OpenTelemetry instrumentation libraries.

Included instrumentation libraries

Requests

Dependencies

Logging

  • ILogger

For more information about ILogger, see Logging in C# and .NET and code examples.

Footnotes

  • ¹: Supports automatic reporting of unhandled/uncaught exceptions
  • ²: Supports OpenTelemetry Metrics
  • ³: By default, logging is only collected at INFO level or higher. To change this setting, see the configuration options.
  • ⁴: By default, logging is only collected when that logging is performed at the WARNING level or higher.

Note

The Azure Monitor OpenTelemetry Distros include custom mapping and logic to automatically emit Application Insights standard metrics.

Tip

All OpenTelemetry metrics whether automatically collected from instrumentation libraries or manual collected from custom coding are currently considered Application Insights "custom metrics" for billing purposes. Learn More.

Add a community instrumentation library

You can collect more data automatically when you include instrumentation libraries from the OpenTelemetry community.

Caution

We don't support or guarantee the quality of community instrumentation libraries. To suggest one for our distro, post or up-vote in our feedback community. Be aware, some are based on experimental OpenTelemetry specs and might introduce future breaking changes.

To add a community library, use the ConfigureOpenTelemetryMeterProvider or ConfigureOpenTelemetryTracerProvider methods, after adding the nuget package for the library.

The following example demonstrates how the Runtime Instrumentation can be added to collect extra metrics.

dotnet add package OpenTelemetry.Instrumentation.Runtime 
// Create a new ASP.NET Core web application builder.
var builder = WebApplication.CreateBuilder(args);

// Configure the OpenTelemetry meter provider to add runtime instrumentation.
builder.Services.ConfigureOpenTelemetryMeterProvider((sp, builder) => builder.AddRuntimeInstrumentation());

// Add the Azure Monitor telemetry service to the application.
// This service will collect and send telemetry data to Azure Monitor.
builder.Services.AddOpenTelemetry().UseAzureMonitor();

// Build the ASP.NET Core web application.
var app = builder.Build();

// Start the ASP.NET Core web application.
app.Run();

Collect custom telemetry

This section explains how to collect custom telemetry from your application.

Depending on your language and signal type, there are different ways to collect custom telemetry, including:

  • OpenTelemetry API
  • Language-specific logging/metrics libraries
  • Application Insights Classic API

The following table represents the currently supported custom telemetry types:

Language Custom Events Custom Metrics Dependencies Exceptions Page Views Requests Traces
ASP.NET Core
   OpenTelemetry API Yes Yes Yes Yes
   ILogger API Yes
   AI Classic API
Java
   OpenTelemetry API Yes Yes Yes Yes
   Logback, Log4j, JUL Yes Yes
   Micrometer Metrics Yes
   AI Classic API Yes Yes Yes Yes Yes Yes Yes
Node.js
   OpenTelemetry API Yes Yes Yes Yes
Python
   OpenTelemetry API Yes Yes Yes Yes
   Python Logging Module Yes
   Events Extension Yes Yes

Note

Application Insights Java 3.x listens for telemetry that's sent to the Application Insights Classic API. Similarly, Application Insights Node.js 3.x collects events created with the Application Insights Classic API. This makes upgrading easier and fills a gap in our custom telemetry support until all custom telemetry types are supported via the OpenTelemetry API.

Add custom metrics

In this context, custom metrics refers to manually instrumenting your code to collect additional metrics beyond what the OpenTelemetry Instrumentation Libraries automatically collect.

The OpenTelemetry API offers six metric "instruments" to cover various metric scenarios and you need to pick the correct "Aggregation Type" when visualizing metrics in Metrics Explorer. This requirement is true when using the OpenTelemetry Metric API to send metrics and when using an instrumentation library.

The following table shows the recommended aggregation types for each of the OpenTelemetry Metric Instruments.

OpenTelemetry Instrument Azure Monitor Aggregation Type
Counter Sum
Asynchronous Counter Sum
Histogram Min, Max, Average, Sum and Count
Asynchronous Gauge Average
UpDownCounter Sum
Asynchronous UpDownCounter Sum

Caution

Aggregation types beyond what's shown in the table typically aren't meaningful.

The OpenTelemetry Specification describes the instruments and provides examples of when you might use each one.

Tip

The histogram is the most versatile and most closely equivalent to the Application Insights GetMetric Classic API. Azure Monitor currently flattens the histogram instrument into our five supported aggregation types, and support for percentiles is underway. Although less versatile, other OpenTelemetry instruments have a lesser impact on your application's performance.

Histogram example

Application startup must subscribe to a Meter by name.

// Create a new ASP.NET Core web application builder.
var builder = WebApplication.CreateBuilder(args);

// Configure the OpenTelemetry meter provider to add a meter named "OTel.AzureMonitor.Demo".
builder.Services.ConfigureOpenTelemetryMeterProvider((sp, builder) => builder.AddMeter("OTel.AzureMonitor.Demo"));

// Add the Azure Monitor telemetry service to the application.
// This service will collect and send telemetry data to Azure Monitor.
builder.Services.AddOpenTelemetry().UseAzureMonitor();

// Build the ASP.NET Core web application.
var app = builder.Build();

// Start the ASP.NET Core web application.
app.Run();

The Meter must be initialized using that same name.

// Create a new meter named "OTel.AzureMonitor.Demo".
var meter = new Meter("OTel.AzureMonitor.Demo");

// Create a new histogram metric named "FruitSalePrice".
Histogram<long> myFruitSalePrice = meter.CreateHistogram<long>("FruitSalePrice");

// Create a new Random object.
var rand = new Random();

// Record a few random sale prices for apples and lemons, with different colors.
myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "green"));
myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));

Counter example

Application startup must subscribe to a Meter by name.

// Create a new ASP.NET Core web application builder.
var builder = WebApplication.CreateBuilder(args);

// Configure the OpenTelemetry meter provider to add a meter named "OTel.AzureMonitor.Demo".
builder.Services.ConfigureOpenTelemetryMeterProvider((sp, builder) => builder.AddMeter("OTel.AzureMonitor.Demo"));

// Add the Azure Monitor telemetry service to the application.
// This service will collect and send telemetry data to Azure Monitor.
builder.Services.AddOpenTelemetry().UseAzureMonitor();

// Build the ASP.NET Core web application.
var app = builder.Build();

// Start the ASP.NET Core web application.
app.Run();

The Meter must be initialized using that same name.

// Create a new meter named "OTel.AzureMonitor.Demo".
var meter = new Meter("OTel.AzureMonitor.Demo");

// Create a new counter metric named "MyFruitCounter".
Counter<long> myFruitCounter = meter.CreateCounter<long>("MyFruitCounter");

// Record the number of fruits sold, grouped by name and color.
myFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
myFruitCounter.Add(2, new("name", "lemon"), new("color", "yellow"));
myFruitCounter.Add(1, new("name", "lemon"), new("color", "yellow"));
myFruitCounter.Add(2, new("name", "apple"), new("color", "green"));
myFruitCounter.Add(5, new("name", "apple"), new("color", "red"));
myFruitCounter.Add(4, new("name", "lemon"), new("color", "yellow"));

Gauge Example

Application startup must subscribe to a Meter by name.

// Create a new ASP.NET Core web application builder.
var builder = WebApplication.CreateBuilder(args);

// Configure the OpenTelemetry meter provider to add a meter named "OTel.AzureMonitor.Demo".
builder.Services.ConfigureOpenTelemetryMeterProvider((sp, builder) => builder.AddMeter("OTel.AzureMonitor.Demo"));

// Add the Azure Monitor telemetry service to the application.
// This service will collect and send telemetry data to Azure Monitor.
builder.Services.AddOpenTelemetry().UseAzureMonitor();

// Build the ASP.NET Core web application.
var app = builder.Build();

// Start the ASP.NET Core web application.
app.Run();

The Meter must be initialized using that same name.

// Get the current process.
var process = Process.GetCurrentProcess();

// Create a new meter named "OTel.AzureMonitor.Demo".
var meter = new Meter("OTel.AzureMonitor.Demo");

// Create a new observable gauge metric named "Thread.State".
// This metric will track the state of each thread in the current process.
ObservableGauge<int> myObservableGauge = meter.CreateObservableGauge("Thread.State", () => GetThreadState(process));

private static IEnumerable<Measurement<int>> GetThreadState(Process process)
{
    // Iterate over all threads in the current process.
    foreach (ProcessThread thread in process.Threads)
    {
        // Create a measurement for each thread, including the thread state, process ID, and thread ID.
        yield return new((int)thread.ThreadState, new("ProcessId", process.Id), new("ThreadId", thread.Id));
    }
}

Add custom exceptions

Select instrumentation libraries automatically report exceptions to Application Insights. However, you might want to manually report exceptions beyond what instrumentation libraries report. For instance, exceptions caught by your code aren't ordinarily reported. You might wish to report them to draw attention in relevant experiences including the failures section and end-to-end transaction views.

  • To log an Exception using an Activity:
    // Start a new activity named "ExceptionExample".
    using (var activity = activitySource.StartActivity("ExceptionExample"))
    {
        // Try to execute some code.
        try
        {
            throw new Exception("Test exception");
        }
        // If an exception is thrown, catch it and set the activity status to "Error".
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error);
            activity?.RecordException(ex);
        }
    }
    
  • To log an Exception using ILogger:
    // Create a logger using the logger factory. The logger category name is used to filter and route log messages.
    var logger = loggerFactory.CreateLogger(logCategoryName);
    
    // Try to execute some code.
    try
    {
        throw new Exception("Test Exception");
    }
    catch (Exception ex)
    {
        // Log an error message with the exception. The log level is set to "Error" and the event ID is set to 0.
        // The log message includes a template and a parameter. The template will be replaced with the value of the parameter when the log message is written.
        logger.Log(
            logLevel: LogLevel.Error,
            eventId: 0,
            exception: ex,
            message: "Hello {name}.",
            args: new object[] { "World" });
    }
    

Add custom spans

You might want to add a custom span in two scenarios. First, when there's a dependency request not already collected by an instrumentation library. Second, when you wish to model an application process as a span on the end-to-end transaction view.

Note

The Activity and ActivitySource classes from the System.Diagnostics namespace represent the OpenTelemetry concepts of Span and Tracer, respectively. You create ActivitySource directly by using its constructor instead of by using TracerProvider. Each ActivitySource class must be explicitly connected to TracerProvider by using AddSource(). That's because parts of the OpenTelemetry tracing API are incorporated directly into the .NET runtime. To learn more, see Introduction to OpenTelemetry .NET Tracing API.

// Define an activity source named "ActivitySourceName". This activity source will be used to create activities for all requests to the application.
internal static readonly ActivitySource activitySource = new("ActivitySourceName");

// Create an ASP.NET Core application builder.
var builder = WebApplication.CreateBuilder(args);

// Configure the OpenTelemetry tracer provider to add a source named "ActivitySourceName". This will ensure that all activities created by the activity source are traced.
builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddSource("ActivitySourceName"));

// Add the Azure Monitor telemetry service to the application. This service will collect and send telemetry data to Azure Monitor.
builder.Services.AddOpenTelemetry().UseAzureMonitor();

// Build the ASP.NET Core application.
var app = builder.Build();

// Map a GET request to the root path ("/") to the specified action.
app.MapGet("/", () =>
{
    // Start a new activity named "CustomActivity". This activity will be traced and the trace data will be sent to Azure Monitor.
    using (var activity = activitySource.StartActivity("CustomActivity"))
    {
        // your code here
    }

    // Return a response message.
    return $"Hello World!";
});

// Start the ASP.NET Core application.
app.Run();

StartActivity defaults to ActivityKind.Internal, but you can provide any other ActivityKind. ActivityKind.Client, ActivityKind.Producer, and ActivityKind.Internal are mapped to Application Insights dependencies. ActivityKind.Server and ActivityKind.Consumer are mapped to Application Insights requests.

Send custom telemetry using the Application Insights Classic API

We recommend you use the OpenTelemetry APIs whenever possible, but there might be some scenarios when you have to use the Application Insights Classic API.

Events
  1. Add Microsoft.ApplicationInsights to your application.

  2. Create a TelemetryClient instance.

Note

It's important to only create once instance of the TelemetryClient per application.

var telemetryConfiguration = new TelemetryConfiguration { ConnectionString = "" };
var telemetryClient = new TelemetryClient(telemetryConfiguration);
  1. Use the client to send custom telemetry.
telemetryClient.TrackEvent("testEvent");

Modify telemetry

This section explains how to modify telemetry.

Add span attributes

These attributes might include adding a custom property to your telemetry. You might also use attributes to set optional fields in the Application Insights schema, like Client IP.

Add a custom property to a Span

Any attributes you add to spans are exported as custom properties. They populate the customDimensions field in the requests, dependencies, traces, or exceptions table.

To add span attributes, use either of the following two ways:

Tip

The advantage of using options provided by instrumentation libraries, when they're available, is that the entire context is available. As a result, users can select to add or filter more attributes. For example, the enrich option in the HttpClient instrumentation library gives users access to the HttpRequestMessage and the HttpResponseMessage itself. They can select anything from it and store it as an attribute.

  1. Many instrumentation libraries provide an enrich option. For guidance, see the readme files of individual instrumentation libraries:

  2. Use a custom processor:

Tip

Add the processor shown here before adding Azure Monitor.

// Create an ASP.NET Core application builder.
var builder = WebApplication.CreateBuilder(args);

// Configure the OpenTelemetry tracer provider to add a new processor named ActivityEnrichingProcessor.
builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddProcessor(new ActivityEnrichingProcessor()));

// Add the Azure Monitor telemetry service to the application. This service will collect and send telemetry data to Azure Monitor.
builder.Services.AddOpenTelemetry().UseAzureMonitor();

// Build the ASP.NET Core application.
var app = builder.Build();

// Start the ASP.NET Core application.
app.Run();

Add ActivityEnrichingProcessor.cs to your project with the following code:

public class ActivityEnrichingProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        // The updated activity will be available to all processors which are called after this processor.
        activity.DisplayName = "Updated-" + activity.DisplayName;
        activity.SetTag("CustomDimension1", "Value1");
        activity.SetTag("CustomDimension2", "Value2");
    }
}

Set the user IP

You can populate the client_IP field for requests by setting an attribute on the span. Application Insights uses the IP address to generate user location attributes and then discards it by default.

Use the add custom property example, but replace the following lines of code in ActivityEnrichingProcessor.cs:

// Add the client IP address to the activity as a tag.
// only applicable in case of activity.Kind == Server
activity.SetTag("client.address", "<IP Address>");

Set the user ID or authenticated user ID

You can populate the user_Id or user_AuthenticatedId field for requests by using the following guidance. User ID is an anonymous user identifier. Authenticated User ID is a known user identifier.

Important

Consult applicable privacy laws before you set the Authenticated User ID.

Use the add custom property example.

// Add the user ID to the activity as a tag, but only if the activity is not null.
activity?.SetTag("enduser.id", "<User Id>");

Add log attributes

OpenTelemetry uses .NET's ILogger. Attaching custom dimensions to logs can be accomplished using a message template.

Filter telemetry

You might use the following ways to filter out telemetry before it leaves your application.

  1. Many instrumentation libraries provide a filter option. For guidance, see the readme files of individual instrumentation libraries:

  2. Use a custom processor:

    Tip

    Add the processor shown here before adding Azure Monitor.

    // Create an ASP.NET Core application builder.
    var builder = WebApplication.CreateBuilder(args);
    
    // Configure the OpenTelemetry tracer provider to add a new processor named ActivityFilteringProcessor.
    builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddProcessor(new ActivityFilteringProcessor()));
    // Configure the OpenTelemetry tracer provider to add a new source named "ActivitySourceName".
    builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddSource("ActivitySourceName"));
    // Add the Azure Monitor telemetry service to the application. This service will collect and send telemetry data to Azure Monitor.
    builder.Services.AddOpenTelemetry().UseAzureMonitor();
    
    // Build the ASP.NET Core application.
    var app = builder.Build();
    
    // Start the ASP.NET Core application.
    app.Run();
    

    Add ActivityFilteringProcessor.cs to your project with the following code:

    public class ActivityFilteringProcessor : BaseProcessor<Activity>
    {
        // The OnStart method is called when an activity is started. This is the ideal place to filter activities.
        public override void OnStart(Activity activity)
        {
            // prevents all exporters from exporting internal activities
            if (activity.Kind == ActivityKind.Internal)
            {
                activity.IsAllDataRequested = false;
            }
        }
    }
    
  3. If a particular source isn't explicitly added by using AddSource("ActivitySourceName"), then none of the activities created by using that source are exported.

Get the trace ID or span ID

You might want to get the trace ID or span ID. If you have logs sent to a destination other than Application Insights, consider adding the trace ID or span ID. Doing so enables better correlation when debugging and diagnosing issues.

Note

The Activity and ActivitySource classes from the System.Diagnostics namespace represent the OpenTelemetry concepts of Span and Tracer, respectively. That's because parts of the OpenTelemetry tracing API are incorporated directly into the .NET runtime. To learn more, see Introduction to OpenTelemetry .NET Tracing API.

// Get the current activity.
Activity activity = Activity.Current;
// Get the trace ID of the activity.
string traceId = activity?.TraceId.ToHexString();
// Get the span ID of the activity.
string spanId = activity?.SpanId.ToHexString();

Next steps

Frequently asked questions

This section provides answers to common questions.

What is OpenTelemetry?

It's a new open-source standard for observability. Learn more at OpenTelemetry.

Why is Microsoft Azure Monitor investing in OpenTelemetry?

Microsoft is among the largest contributors to OpenTelemetry.

The key value propositions of OpenTelemetry are that it's vendor-neutral and provides consistent APIs/SDKs across languages.

Over time, we believe OpenTelemetry will enable Azure Monitor customers to observe applications written in languages beyond our supported languages. It also expands the types of data you can collect through a rich set of instrumentation libraries. Furthermore, OpenTelemetry SDKs tend to be more performant at scale than their predecessors, the Application Insights SDKs.

Finally, OpenTelemetry aligns with Microsoft's strategy to embrace open source.

What's the status of OpenTelemetry?

See OpenTelemetry Status.

What is the "Azure Monitor OpenTelemetry Distro"?

You can think of it as a thin wrapper that bundles together all the OpenTelemetry components for a first class experience on Azure. This wrapper is also called a distribution in OpenTelemetry.

Why should I use the "Azure Monitor OpenTelemetry Distro"?

There are several advantages to using the Azure Monitor OpenTelemetry Distro over native OpenTelemetry from the community:

In the spirit of OpenTelemetry, we designed the distro to be open and extensible. For example, you can add:

  • An OpenTelemetry Protocol (OTLP) exporter and send to a second destination simultaneously
  • Other instrumentation libraries not included in the distro

Because the Distro provides an OpenTelemetry distribution, the Distro supports anything supported by OpenTelemetry. For example, you can add more telemetry processors, exporters, or instrumentation libraries, if OpenTelemetry supports them.

Note

The Distro sets the sampler to a custom, fixed-rate sampler for Application Insights. You can change this to a different sampler, but doing so might disable some of the Distro's included capabilities. For more information about the supported sampler, see the Enable Sampling section of Configure Azure Monitor OpenTelemetry.

For languages without a supported standalone OpenTelemetry exporter, the Azure Monitor OpenTelemetry Distro is the only currently supported way to use OpenTelemetry with Azure Monitor. For languages with a supported standalone OpenTelemetry exporter, you have the option of using either the Azure Monitor OpenTelemetry Distro or the appropriate standalone OpenTelemetry exporter depending on your telemetry scenario. For more information, see When should I use the Azure Monitor OpenTelemetry exporter?.

How can I test out the Azure Monitor OpenTelemetry Distro?

Check out our enablement docs for .NET, Java, JavaScript (Node.js), and Python.

Should I use OpenTelemetry or the Application Insights SDK?

We recommend using the OpenTelemetry Distro unless you require a feature that is only available with formal support in the Application Insights SDK.

Adopting OpenTelemetry now prevents having to migrate at a later date.

When should I use the Azure Monitor OpenTelemetry exporter?

For ASP.NET Core, Java, Node.js, and Python, we recommend using the Azure Monitor OpenTelemetry Distro. It's one line of code to get started.

For all other .NET scenarios, including classic ASP.NET, console apps, etc., we recommend using the .NET Azure Monitor OpenTelemetry exporter: Azure.Monitor.OpenTelemetry.Exporter.

For more complex Python telemetry scenarios that require advanced configuration, we recommend using the Python Azure Monitor OpenTelemetry Exporter.

What's the current release state of features within the Azure Monitor OpenTelemetry Distro?

The following chart breaks out OpenTelemetry feature support for each language.

Feature .NET Node.js Python Java
Distributed tracing
Custom metrics
Standard metrics (accuracy currently affected by sampling)
Fixed-rate sampling
Offline storage and automatic retries
Exception reporting
Logs collection ⚠️
Custom Events ⚠️ ⚠️ ⚠️
Microsoft Entra authentication
Live metrics
Detect Resource Context for VM/VMSS and App Service
Detect Resource Context for AKS and Functions
Availability Testing Span Filtering
Autopopulation of user ID, authenticated user ID, and user IP
Manually override/set operation name, user ID, or authenticated user ID
Adaptive sampling
Profiler ⚠️
Snapshot Debugger

Key

Can OpenTelemetry be used for web browsers?

Yes, but we don't recommend it and Azure doesn't support it. OpenTelemetry JavaScript is heavily optimized for Node.js. Instead, we recommend using the Application Insights JavaScript SDK.

When can we expect the OpenTelemetry SDK to be available for use in web browsers?

The OpenTelemetry web SDK doesn't have a determined availability timeline. We're likely several years away from a browser SDK that is a viable alternative to the Application Insights JavaScript SDK.

Can I test OpenTelemetry in a web browser today?

The OpenTelemetry web sandbox is a fork designed to make OpenTelemetry work in a browser. It's not yet possible to send telemetry to Application Insights. The SDK doesn't define general client events.

Is running Application Insights alongside competitor agents like AppDynamics, DataDog, and NewRelic supported?

No. This practice isn't something we plan to test or support, although our Distros allow you to export to an OTLP endpoint alongside Azure Monitor simultaneously.

Can I use preview features in production environments?

We don't recommend it. See Supplemental terms of use for Microsoft Azure previews.

What's the difference between manual and automatic instrumentation?

See the OpenTelemetry Overview.

Can I use the OpenTelemetry Collector?

Some customers use the OpenTelemetry Collector as an agent alternative, even though Microsoft doesn't officially support an agent-based approach for application monitoring yet. In the meantime, the open-source community contributed an OpenTelemetry Collector Azure Monitor Exporter that some customers are using to send data to Azure Monitor Application Insights. This is not supported by Microsoft.

What's the difference between OpenCensus and OpenTelemetry?

OpenCensus is the precursor to OpenTelemetry. Microsoft helped bring together OpenTracing and OpenCensus to create OpenTelemetry, a single observability standard for the world. The current production-recommended Python SDK for Azure Monitor is based on OpenCensus. Microsoft is committed to making Azure Monitor based on OpenTelemetry.

Troubleshooting

Not working? Check out the troubleshooting page for ASP.NET Core.

Support

Select a tab for the language of your choice to discover support options.

OpenTelemetry feedback

To provide feedback: