Edit

Share via

Add and modify Azure Monitor OpenTelemetry for .NET, Java, Node.js, and Python applications

  • Article

This guide provides instructions on integrating and customizing OpenTelemetry (OTel) instrumentation within Azure Monitor Application Insights.

To learn more about OpenTelemetry concepts, see the OpenTelemetry overview or OpenTelemetry FAQ.

Automatic data collection

The distros automatically collect data by bundling OpenTelemetry instrumentation libraries.

Included instrumentation libraries

Requests

Dependencies

Logs

Examples of using the Python logging library can be found on GitHub.

Telemetry emitted by Azure SDKS is automatically collected by default.

Footnotes

  • ¹: Supports automatic reporting of unhandled/uncaught exceptions
  • ²: Supports OpenTelemetry Metrics
  • ³: By default, logging is only collected at INFO level or higher. To change this setting, see the configuration options.
  • ⁴: By default, logging is only collected when that logging is performed at the WARNING level or higher.

Note

The Azure Monitor OpenTelemetry Distros include custom mapping and logic to automatically emit Application Insights standard metrics.

Tip

All OpenTelemetry metrics whether automatically collected from instrumentation libraries or manually collected from custom coding are currently considered Application Insights "custom metrics" for billing purposes. Learn more.

Add a community instrumentation library

You can collect more data automatically when you include instrumentation libraries from the OpenTelemetry community.

Caution

We don't support or guarantee the quality of community instrumentation libraries. To suggest one for our distro, post or up-vote in our feedback community. Be aware, some are based on experimental OpenTelemetry specs and might introduce future breaking changes.

To add a community instrumentation library (not officially supported/included in Azure Monitor distro), you can instrument directly with the instrumentations. The list of community instrumentation libraries can be found here.

Note

Instrumenting a supported instrumentation library manually with instrument() in conjunction with the distro configure_azure_monitor() is not recommended. This is not a supported scenario and you may get undesired behavior for your telemetry.

Python 
# Import the `configure_azure_monitor()`, `SQLAlchemyInstrumentor`, `create_engine`, and `text` functions from the appropriate packages.
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from sqlalchemy import create_engine, text

# Configure OpenTelemetry to use Azure Monitor.
configure_azure_monitor()

# Create a SQLAlchemy engine.
engine = create_engine("sqlite:///:memory:")

# SQLAlchemy instrumentation is not officially supported by this package, however, you can use the OpenTelemetry `instrument()` method manually in conjunction with `configure_azure_monitor()`.
SQLAlchemyInstrumentor().instrument(
    engine=engine,
)

# Database calls using the SQLAlchemy library will be automatically captured.
with engine.connect() as conn:
    result = conn.execute(text("select 'hello world'"))
    print(result.all())

Collect custom telemetry

This section explains how to collect custom telemetry from your application.

Depending on your language and signal type, there are different ways to collect custom telemetry, including:

  • OpenTelemetry API
  • Language-specific logging/metrics libraries
  • Application Insights Classic API

The following table represents the currently supported custom telemetry types:

Language Custom Events Custom Metrics Dependencies Exceptions Page Views Requests Traces
ASP.NET Core
   OpenTelemetry API Yes Yes Yes Yes
   ILogger API Yes
   AI Classic API
Java
   OpenTelemetry API Yes Yes Yes Yes
   Logback, Log4j, JUL Yes Yes
   Micrometer Metrics Yes
   AI Classic API Yes Yes Yes Yes Yes Yes Yes
Node.js
   OpenTelemetry API Yes Yes Yes Yes
Python
   OpenTelemetry API Yes Yes Yes Yes
   Python Logging Module Yes
   Events Extension Yes Yes

Note

Application Insights Java 3.x listens for telemetry that's sent to the Application Insights Classic API. Similarly, Application Insights Node.js 3.x collects events created with the Application Insights Classic API. This makes upgrading easier and fills a gap in our custom telemetry support until all custom telemetry types are supported via the OpenTelemetry API.

Add custom metrics

In this context, the custom metrics term refers to manually instrumenting your code to collect additional metrics beyond what the OpenTelemetry Instrumentation Libraries automatically collect.

The OpenTelemetry API offers six metric "instruments" to cover various metric scenarios and you need to pick the correct "Aggregation Type" when visualizing metrics in Metrics Explorer. This requirement is true when using the OpenTelemetry Metric API to send metrics and when using an instrumentation library.

The following table shows the recommended aggregation types for each of the OpenTelemetry Metric Instruments.

OpenTelemetry Instrument Azure Monitor Aggregation Type
Counter Sum
Asynchronous Counter Sum
Histogram Min, Max, Average, Sum, and Count
Asynchronous Gauge Average
UpDownCounter Sum
Asynchronous UpDownCounter Sum

Caution

Aggregation types beyond what's shown in the table typically aren't meaningful.

The OpenTelemetry Specification describes the instruments and provides examples of when you might use each one.

Tip

The histogram is the most versatile and most closely equivalent to the Application Insights GetMetric Classic API. Azure Monitor currently flattens the histogram instrument into our five supported aggregation types, and support for percentiles is underway. Although less versatile, other OpenTelemetry instruments have a lesser impact on your application's performance.

Histogram example

Python 
# Import the `configure_azure_monitor()` and `metrics` functions from the appropriate packages.
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import metrics

import os

# Configure OpenTelemetry to use Azure Monitor with the specified connection string.
# Replace `<your-connection-string>` with the connection string to your Azure Monitor Application Insights resource.
configure_azure_monitor(
    connection_string="<your-connection-string>",
)

# Opt in to allow grouping of your metrics via a custom metrics namespace in app insights metrics explorer.
# Specify the namespace name using get_meter("namespace-name")
os.environ["APPLICATIONINSIGHTS_METRIC_NAMESPACE_OPT_IN"] = "true"

# Get a meter provider and a meter with the name "otel_azure_monitor_histogram_demo".
meter = metrics.get_meter_provider().get_meter("otel_azure_monitor_histogram_demo")

# Record three values to the histogram.
histogram = meter.create_histogram("histogram")
histogram.record(1.0, {"test_key": "test_value"})
histogram.record(100.0, {"test_key2": "test_value"})
histogram.record(30.0, {"test_key": "test_value2"})

# Wait for background execution.
input()

Counter example

Python 
# Import the `configure_azure_monitor()` and `metrics` functions from the appropriate packages.
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import metrics

import os

# Configure OpenTelemetry to use Azure Monitor with the specified connection string.
# Replace `<your-connection-string>` with the connection string to your Azure Monitor Application Insights resource.
configure_azure_monitor(
    connection_string="<your-connection-string>",
)

# Opt in to allow grouping of your metrics via a custom metrics namespace in app insights metrics explorer.
# Specify the namespace name using get_meter("namespace-name")
os.environ["APPLICATIONINSIGHTS_METRIC_NAMESPACE_OPT_IN"] = "true"

# Get a meter provider and a meter with the name "otel_azure_monitor_counter_demo".
meter = metrics.get_meter_provider().get_meter("otel_azure_monitor_counter_demo")

# Create a counter metric with the name "counter".
counter = meter.create_counter("counter")

# Add three values to the counter.
# The first argument to the `add()` method is the value to add.
# The second argument is a dictionary of dimensions.
# Dimensions are used to group related metrics together.
counter.add(1.0, {"test_key": "test_value"})
counter.add(5.0, {"test_key2": "test_value"})
counter.add(3.0, {"test_key": "test_value2"})

# Wait for background execution.
input()

Gauge example

Python 
# Import the necessary packages.
from typing import Iterable
import os

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import metrics
from opentelemetry.metrics import CallbackOptions, Observation

# Configure OpenTelemetry to use Azure Monitor with the specified connection string.
# Replace `<your-connection-string>` with the connection string to your Azure Monitor Application Insights resource.
configure_azure_monitor(
    connection_string="<your-connection-string>",
)

# Opt in to allow grouping of your metrics via a custom metrics namespace in app insights metrics explorer.
# Specify the namespace name using get_meter("namespace-name")
os.environ["APPLICATIONINSIGHTS_METRIC_NAMESPACE_OPT_IN"] = "true"

# Get a meter provider and a meter with the name "otel_azure_monitor_gauge_demo".
meter = metrics.get_meter_provider().get_meter("otel_azure_monitor_gauge_demo")

# Define two observable gauge generators.
# The first generator yields a single observation with the value 9.
# The second generator yields a sequence of 10 observations with the value 9 and a different dimension value for each observation.
def observable_gauge_generator(options: CallbackOptions) -> Iterable[Observation]:
    yield Observation(9, {"test_key": "test_value"})

def observable_gauge_sequence(options: CallbackOptions) -> Iterable[Observation]:
    observations = []
    for i in range(10):
        observations.append(
            Observation(9, {"test_key": i})
        )
    return observations

# Create two observable gauges using the defined generators.
gauge = meter.create_observable_gauge("gauge", [observable_gauge_generator])
gauge2 = meter.create_observable_gauge("gauge2", [observable_gauge_sequence])

# Wait for background execution.
input()

Add custom exceptions

Select instrumentation libraries automatically report exceptions to Application Insights. However, you might want to manually report exceptions beyond what instrumentation libraries report. For instance, exceptions caught by your code aren't ordinarily reported. You might wish to report them to draw attention in relevant experiences including the failures section and end-to-end transaction views.

The OpenTelemetry Python SDK is implemented in such a way that exceptions thrown are automatically captured and recorded. See the following code sample for an example of this behavior:

Python 
# Import the necessary packages.
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

# Configure OpenTelemetry to use Azure Monitor with the specified connection string.
# Replace `<your-connection-string>` with the connection string to your Azure Monitor Application Insights resource.
configure_azure_monitor(
    connection_string="<your-connection-string>",
)

# Get a tracer for the current module.
tracer = trace.get_tracer("otel_azure_monitor_exception_demo")

# Exception events
try:
    # Start a new span with the name "hello".
    with tracer.start_as_current_span("hello") as span:
        # This exception will be automatically recorded
        raise Exception("Custom exception message.")
except Exception:
    print("Exception raised")

If you would like to record exceptions manually, you can disable that option within the context manager and use record_exception() directly as shown in the following example:

Python 
...
# Start a new span with the name "hello" and disable exception recording.
with tracer.start_as_current_span("hello", record_exception=False) as span:
    try:
        # Raise an exception.
        raise Exception("Custom exception message.")
    except Exception as ex:
        # Manually record exception
        span.record_exception(ex)
...

Add custom spans

You might want to add a custom span in two scenarios. First, when there's a dependency request not already collected by an instrumentation library. Second, when you wish to model an application process as a span on the end-to-end transaction view.

The OpenTelemetry API can be used to add your own spans, which appear in the requests and dependencies tables in Application Insights.

The code example shows how to use the tracer.start_as_current_span() method to start, make the span current, and end the span within its context.

Python 
...
# Import the necessary packages.
from opentelemetry import trace

# Get a tracer for the current module.
tracer = trace.get_tracer(__name__)

# Start a new span with the name "my first span" and make it the current span.
# The "with" context manager starts, makes the span current, and ends the span within it's context
with tracer.start_as_current_span("my first span") as span:
    try:
        # Do stuff within the context of this span.
        # All telemetry generated within this scope will be attributed to this span.
    except Exception as ex:
        # Record the exception on the span.
        span.record_exception(ex)
...

By default, the span is in the dependencies table with a dependency type of InProc.

If your method represents a background job not already captured by autoinstrumentation, we recommend setting the attribute kind = SpanKind.SERVER to ensure it appears in the Application Insights requests table.

Python 
...
# Import the necessary packages.
from opentelemetry import trace
from opentelemetry.trace import SpanKind

# Get a tracer for the current module.
tracer = trace.get_tracer(__name__)

# Start a new span with the name "my request span" and the kind set to SpanKind.SERVER.
with tracer.start_as_current_span("my request span", kind=SpanKind.SERVER) as span:
    # Do stuff within the context of this span.
...

Send custom telemetry using the Application Insights Classic API

We recommend you use the OpenTelemetry APIs whenever possible, but there might be some scenarios when you have to use the Application Insights Classic API.

Unlike other languages, Python doesn't have an Application Insights SDK. You can meet all your monitoring needs with the Azure Monitor OpenTelemetry Distro, except for sending customEvents. Until the OpenTelemetry Events API stabilizes, use the Azure Monitor Events Extension with the Azure Monitor OpenTelemetry Distro to send customEvents to Application Insights.

Install the distro and the extension:

Console 
pip install azure-monitor-opentelemetry
pip install azure-monitor-events-extension

Use the track_event API offered in the extension to send customEvents:

Python 
...
from azure.monitor.events.extension import track_event
from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor()

# Use the track_event() api to send custom event telemetry
# Takes event name and custom dimensions
track_event("Test event", {"key1": "value1", "key2": "value2"})

input()
...

Modify telemetry

This section explains how to modify telemetry.

Add span attributes

These attributes might include adding a custom property to your telemetry. You might also use attributes to set optional fields in the Application Insights schema, like Client IP.

Add a custom property to a Span

Any attributes you add to spans are exported as custom properties. They populate the customDimensions field in the requests, dependencies, traces, or exceptions table.

Use a custom processor:

Python 
...
# Import the necessary packages.
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

# Create a SpanEnrichingProcessor instance.
span_enrich_processor = SpanEnrichingProcessor()

# Configure OpenTelemetry to use Azure Monitor with the specified connection string.
# Replace `<your-connection-string>` with the connection string to your Azure Monitor Application Insights resource.
configure_azure_monitor(
    connection_string="<your-connection-string>",
    # Configure the custom span processors to include span enrich processor.
    span_processors=[span_enrich_processor],
)

...

Add SpanEnrichingProcessor to your project with the following code:

Python 
# Import the SpanProcessor class from the opentelemetry.sdk.trace module.
from opentelemetry.sdk.trace import SpanProcessor

class SpanEnrichingProcessor(SpanProcessor):

    def on_end(self, span):
        # Prefix the span name with the string "Updated-".
        span._name = "Updated-" + span.name
        # Add the custom dimension "CustomDimension1" with the value "Value1".
        span._attributes["CustomDimension1"] = "Value1"
         # Add the custom dimension "CustomDimension2" with the value "Value2".
        span._attributes["CustomDimension2"] = "Value2"

Set the user IP

You can populate the client_IP field for requests by setting an attribute on the span. Application Insights uses the IP address to generate user location attributes and then discards it by default.

Use the custom property example, but replace the following lines of code in SpanEnrichingProcessor.py:

Python 
# Set the `http.client_ip` attribute of the span to the specified IP address.
span._attributes["http.client_ip"] = "<IP Address>"

Set the user ID or authenticated user ID

You can populate the user_Id or user_AuthenticatedId field for requests by using the following guidance. User ID is an anonymous user identifier. Authenticated User ID is a known user identifier.

Important

Consult applicable privacy laws before you set the Authenticated User ID.

Use the custom property example, but replace the following lines of code:

Python 
# Set the `enduser.id` attribute of the span to the specified user ID.
span._attributes["enduser.id"] = "<User ID>"

Add log attributes

The Python logging library is autoinstrumented. You can attach custom dimensions to your logs by passing a dictionary into the extra argument of your logs:

Python 
...
# Create a warning log message with the properties "key1" and "value1".
logger.warning("WARNING: Warning log with properties", extra={"key1": "value1"})
...

Get the trace ID or span ID

You can obtain the Trace ID and Span ID of the currently active Span using following steps.

Get the request trace ID and the span ID in your code:

Python 
# Import the necessary libraries.
from opentelemetry import trace

# Get the trace ID and span ID of the current span.
trace_id = trace.get_current_span().get_span_context().trace_id
span_id = trace.get_current_span().get_span_context().span_id

Next steps