Data Collection Basics of Azure Monitor Application Insights

Before you can monitor your application, it needs to be instrumented.

In the following sections, we cover some data collection basics of Azure Monitor Application Insights.

Instrumentation Options

At a basic level, "instrumenting" is simply enabling an application to capture telemetry.

There are two methods to instrument your application:

• Automatic instrumentation (autoinstrumentation)
• Manual instrumentation

Autoinstrumentation enables telemetry collection through configuration without touching the application's code. Although it's more convenient, it tends to be less configurable. It's also not available in all languages. See Autoinstrumentation supported environments and languages. When autoinstrumentation is available, it's the easiest way to enable Azure Monitor Application Insights.

Manual instrumentation is coding against the Application Insights or OpenTelemetry API. In the context of a user, it typically refers to installing a language-specific SDK in an application. This means that you have to manage the updates to the latest package version by yourself. You can use this option if you need to make custom dependency calls or API calls that are not captured by default with autoinstrumentation. There are two options for manual instrumentation:

• Application Insights SDKs
• Azure Monitor OpenTelemetry Distros.

While we see OpenTelemetry as our future direction, we have no plans to stop collecting data from older SDKs. We still have a way to go before our Azure OpenTelemetry Distros reach feature parity with our Application Insights SDKs. In many cases, customers continue to choose to use Application Insights SDKs for quite some time.

Important

"Manual" doesn't mean you'll be required to write complex code to define spans for distributed traces, although it remains an option. Instrumentation Libraries packaged into our Distros enable you to effortlessly capture telemetry signals across common frameworks and libraries. We're actively working to instrument the most popular Azure Service SDKs using OpenTelemetry so these signals are available to customers who use the Azure Monitor OpenTelemetry Distro.

Telemetry Types

Telemetry, the data collected to observe your application, can be broken into three types or "pillars":

• Distributed Tracing
• Metrics
• Logs

A complete observability story includes all three pillars, and Application Insights further breaks down these pillars into tables based on our data model. Our Application Insights SDKs or Azure Monitor OpenTelemetry Distros include everything you need to power Application Performance Monitoring on Azure. The package itself is free to install, and you only pay for the data you ingest in Azure Monitor.

The following sources explain the three pillars:

• OpenTelemetry community website
• OpenTelemetry specifications
• Distributed Systems Observability by Cindy Sridharan

Telemetry Routing

There are two ways to send your data to Azure Monitor (or any vendor):

• Via a direct exporter
• Via an agent

A direct exporter sends telemetry in-process (from the application's code) directly to the Azure Monitor ingestion endpoint. The main advantage of this approach is onboarding simplicity.

The currently available Application Insights SDKs and Azure Monitor OpenTelemetry Distros rely on a direct exporter.

Note

For Azure Monitor's position on the OpenTelemetry-Collector, see the OpenTelemetry FAQ.

Tip

If you are planning to use OpenTelemetry-Collector for sampling or additional data processing, you may be able to get these same capabilities built-in to Azure Monitor. Customers who have migrated to Workspace-based Application Insights can benefit from Ingestion-time Transformations. To enable, follow the details in the tutorial, skipping the step that shows how to set up a diagnostic setting since with Workspace-centric Application Insights this is already configured. If you’re filtering less than 50% of the overall volume, it’s no additional cost. After 50%, there is a cost but much less than the standard per GB charge.

OpenTelemetry

Microsoft is excited to embrace OpenTelemetry as the future of telemetry instrumentation. You, our customers, asked for vendor-neutral instrumentation, and we're pleased to partner with the OpenTelemetry community to create consistent APIs and SDKs across languages.

Microsoft worked with project stakeholders from two previously popular open-source telemetry projects, OpenCensus and OpenTracing. Together, we helped to create a single project, OpenTelemetry. OpenTelemetry includes contributions from all major cloud and Application Performance Management (APM) vendors and lives within the Cloud Native Computing Foundation (CNCF). Microsoft is a Platinum Member of the CNCF.

For terminology, see the glossary in the OpenTelemetry specifications.

Some legacy terms in Application Insights are confusing because of the industry convergence on OpenTelemetry. The following table highlights these differences. OpenTelemetry terms are replacing Application Insights terms.

Application Insights	OpenTelemetry
Autocollectors	Instrumentation libraries
Channel	Exporter
Codeless / Agent-based	Autoinstrumentation
Traces	Logs
Requests	Server Spans
Dependencies	Other Span Types (Client, Internal, etc.)
Operation ID	Trace ID
ID or Operation Parent ID	Span ID

Troubleshooting

ASP.NET Core

Step 1: Enable diagnostic logging

The Azure Monitor Exporter uses EventSource for its internal logging. The exporter logs are available to any EventListener by opting in to the source that's named OpenTelemetry-AzureMonitor-Exporter. For troubleshooting steps, see OpenTelemetry Troubleshooting on GitHub.

Step 2: Test connectivity between your application host and the ingestion service

Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.

Known issues

The following items are known issues for the Azure Monitor OpenTelemetry Exporters:

• The operation name is missing from dependency telemetry. The missing operation name causes failures and adversely affects performance tab experience.

• The device model is missing from request and dependency telemetry. The missing device model adversely affects device cohort analysis.

.NET

Step 1: Enable diagnostic logging

The Azure Monitor Exporter uses EventSource for its internal logging. The exporter logs are available to any EventListener by opting in to the source that's named OpenTelemetry-AzureMonitor-Exporter. For troubleshooting steps, see OpenTelemetry Troubleshooting on GitHub.

Step 2: Test connectivity between your application host and the ingestion service

Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.

Known issues

The following items are known issues for the Azure Monitor OpenTelemetry Exporters:

• The operation name is missing from dependency telemetry. The missing operation name causes failures and adversely affects performance tab experience.

• The device model is missing from request and dependency telemetry. The missing device model adversely affects device cohort analysis.

Java

Step 1: Enable diagnostic logging

By default, diagnostic logging is enabled in Azure Monitor Application Insights. For more information, see Troubleshoot guide: Azure Monitor Application Insights for Java.

Step 2: Test connectivity between your application host and the ingestion service

Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.

Known issues

• If you download the Application Insights client library for installation from a browser, sometimes the downloaded JAR file is corrupted and is about half the size of the source file. If you experience this problem, download the JAR file by running the curl or wget command, as shown in the following example command calls:

  curl --location --output applicationinsights-agent-3.4.11.jar https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.4.11/applicationinsights-agent-3.4.11.jar
  

  wget --output-document=applicationinsights-agent-3.4.11.jar https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.4.11/applicationinsights-agent-3.4.11.jar
  

  Note

  The example command calls apply to Application Insights for Java version 3.4.11. To find the version number and URL address of the current release of Application Insights for Java, see https://github.com/microsoft/ApplicationInsights-Java/releases.

Java native

The following steps are applicable to Spring Boot native applications.

Step 1: Verify the OpenTelemetry version

You might notice the following message during the application startup:

WARN  c.a.m.a.s.OpenTelemetryVersionCheckRunner - The OpenTelemetry version is not compatible with the spring-cloud-azure-starter-monitor dependency.
The OpenTelemetry version should be <version>

In this case, you have to import the OpenTelemetry Bills of Materials by following the OpenTelemetry documentation in the Spring Boot starter.

Step 2: Enable self-diagnostics

If something doesn't work as expected, you can enable self-diagnostics at the DEBUG level to get some insights. To do so, set the self-diagnostics level to ERROR, WARN, INFO, DEBUG, or TRACE by using the APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL environment variable.

To enable self-diagnostics at the DEBUG level when running a docker container, run the following command:

docker run -e APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL=DEBUG <image-name>

Note

Replace <image-name> with the docker image name accordingly.

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Node.js

Step 1: Enable diagnostic logging

Azure Monitor Exporter uses the OpenTelemetry API logger for internal logs. To enable the logger, run the following code snippet:

const { diag, DiagConsoleLogger, DiagLogLevel } = require("@opentelemetry/api");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");

const provider = new NodeTracerProvider();
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ALL);
provider.register();

Step 2: Test connectivity between your application host and the ingestion service

Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.

Known issues

The following items are known issues for the Azure Monitor OpenTelemetry Exporters:

• The operation name is missing from dependency telemetry. The missing operation name causes failures and adversely affects performance tab experience.

• The device model is missing from request and dependency telemetry. The missing device model adversely affects device cohort analysis.

• The database server name is missing from the dependency name. Because the database server name isn't included, OpenTelemetry Exporters incorrectly aggregate tables that have the same name onto different servers.

Python

Step 1: Enable diagnostic logging

The Microsoft Azure Monitor Exporter uses the Python standard logging library for its internal logging. OpenTelemetry API and Azure Monitor Exporter logs are assigned a severity level of WARNING or ERROR for irregular activity. The INFO severity level is used for regular or successful activity.

By default, the Python logging library sets the severity level to WARNING. Therefore, you must change the severity level to see logs under this severity setting. The following example code shows how to output logs of all severity levels to the console and a file:

...
import logging

logging.basicConfig(format = "%(asctime)s:%(levelname)s:%(message)s", level = logging.DEBUG)

logger = logging.getLogger(__name__)
file = logging.FileHandler("example.log")
stream = logging.StreamHandler()
logger.addHandler(file)
logger.addHandler(stream)
...

Step 2: Test connectivity between your application host and the ingestion service

Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.

Step 3: Avoid duplicate telemetry

Duplicate telemetry is often caused if you create multiple instances of processors or exporters. Make sure that you run only one exporter and processor at a time for each telemetry pillar (logs, metrics, and distributed tracing).

The following sections describe scenarios that can cause duplicate telemetry.

Duplicate trace logs in Azure Functions

If you see a pair of entries for each trace log within Application Insights, you probably enabled the following types of logging instrumentation:

• The native logging instrumentation in Azure Functions
• The azure-monitor-opentelemetry logging instrumentation within the distribution

To prevent duplication, you can disable the distribution's logging, but leave the native logging instrumentation in Azure Functions enabled. To do this, set the OTEL_LOGS_EXPORTER environment variable to None.

Duplicate telemetry in "Always On" Azure Functions

If the Always On setting in Azure Functions is set to On, Azure Functions keeps some processes running in the background after each run is complete. For instance, suppose that you have a five-minute timer function that calls configure_azure_monitor each time. After 20 minutes, you then might have four metric exporters that are running at the same time. This situation might be the source of your duplicate metrics telemetry.

In this situation, either set the Always On setting to Off, or try manually shutting down the providers between each configure_azure_monitor call. To shut down each provider, run shutdown calls for each current meter, tracer, and logger provider, as shown in the following code:

get_meter_provider().shutdown()
get_tracer_provider().shutdown()
get_logger_provider().shutdown()

Azure Workbooks and Jupyter Notebooks

Azure Workbooks and Jupyter Notebooks might keep exporter processes running in the background. To prevent duplicate telemetry, clear the cache before you make more calls to configure_azure_monitor.

Step 4: Make sure that Flask request data is collected

If you implement a Flask application, you might find that you can't collect Requests table data from Application Insights while you use the Azure Monitor OpenTelemetry Distro client library for Python. This issue could occur if you don't structure your import declarations correctly. You might be importing the flask.Flask web application framework before you call the configure_azure_monitor function to instrument the Flask library. For example, the following code doesn't successfully instrument the Flask app:

from azure.monitor.opentelemetry import configure_azure_monitor
from flask import Flask

configure_azure_monitor()

app = Flask(__name__)

Instead, we recommend that you import the flask module as a whole, and then call configure_azure_monitor to configure OpenTelemetry to use Azure Monitor before you access flask.Flask:

from azure.monitor.opentelemetry import configure_azure_monitor
import flask

configure_azure_monitor()

app = flask.Flask(__name__)

Alternatively, you can call configure_azure_monitor before you import flask.Flask:

from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor()

from flask import Flask

app = Flask(__name__)

Support

Select a tab for the language of your choice to discover support options.

ASP.NET Core

• For Azure support issues, open an Azure support ticket.
• For OpenTelemetry issues, contact the OpenTelemetry .NET community directly.
• For a list of open issues related to Azure Monitor Exporter, see the GitHub Issues Page.

.NET

• For Azure support issues, open an Azure support ticket.
• For OpenTelemetry issues, contact the OpenTelemetry .NET community directly.
• For a list of open issues related to Azure Monitor Exporter, see the GitHub Issues Page.

Java

• For Azure support issues, open an Azure support ticket.
• For help with troubleshooting, review the troubleshooting steps.
• For OpenTelemetry issues, contact the OpenTelemetry community directly.
• For a list of open issues related to Azure Monitor Java Autoinstrumentation, see the GitHub Issues Page.

Java native

• For Azure support issues, open an Azure support ticket.
• For OpenTelemetry issues, contact the OpenTelemetry community directly.
• For a list of open issues with Spring Boot native applications, see the GitHub Issues Page.
• For a list of open issues with Quarkus native applications, see the GitHub Issues Page.

Node.js

• For Azure support issues, open an Azure support ticket.
• For OpenTelemetry issues, contact the OpenTelemetry JavaScript community directly.
• For a list of open issues related to Azure Monitor Exporter, see the GitHub Issues Page.

Python

• For Azure support issues, open an Azure support ticket.
• For OpenTelemetry issues, contact the OpenTelemetry Python community directly.
• For a list of open issues related to Azure Monitor Distro, see the GitHub Issues Page.

OpenTelemetry feedback

To provide feedback:

• Fill out the OpenTelemetry community's customer feedback survey.
• Tell Microsoft about yourself by joining the OpenTelemetry Early Adopter Community.
• Engage with other Azure Monitor users in the Microsoft Tech Community.
• Make a feature request at the Azure Feedback Forum.

Frequently asked questions

Where can I find a list of Application Insights SDK versions and their names?

A list of SDK versions and names is hosted on GitHub. For more information, see SDK Version.

Next steps

Select your enablement approach:

• Autoinstrumentation
• Application Insights SDKs
  • ASP.NET
  • ASP.NET Core
  • Node.js
  • Python
  • JavaScript: Web
• Azure Monitor OpenTelemetry Distro

Check out the Azure Monitor Application Insights FAQ and OpenTelemetry FAQ for more information.