Events
Mar 17, 9 PM - Mar 21, 10 AM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register nowThis browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Using an OpenTelemetry data agent with your Azure Container Apps environment, you can choose to send observability data in an OpenTelemetry format by:
Piping data from an agent into a desired endpoint. Destination options include Azure Monitor Application Insights, Datadog, and any OpenTelemetry Protocol (OTLP)-compatible endpoint.
Easily changing destination endpoints without having to reconfigure how they emit data, and without having to manually run an OpenTelemetry agent.
This article shows you how to set up and configure an OpenTelemetry agent for your container app.
OpenTelemetry agents live within your container app environment. You configure agent settings via an ARM template or Bicep calls to the environment, or through the CLI, or through Terraform (via the AzAPI provider).
Each endpoint type (Azure Monitor Application Insights, DataDog, and OTLP) has specific configuration requirements.
Enabling the managed OpenTelemetry agent to your environment doesn't automatically mean the agent collects data. Agents only send data based on your configuration settings and instrumenting your code correctly.
Prepare your application to collect data by installing the OpenTelemetry SDK and follow the OpenTelemetry guidelines to instrument metrics, logs, or traces.
Before you can send data to a collection destination, you first need to create an instance of the destination service. For example, if you want to send data to Azure Monitor Application Insights, you need to create an Application Insights instance ahead of time.
The managed OpenTelemetry agent accepts the following destinations:
The following table shows you what type of data you can send to each destination:
Destination | Logs | Metrics | Traces |
---|---|---|---|
Azure App Insights | Yes | No | Yes |
Datadog | No | Yes | Yes |
OpenTelemetry protocol (OTLP) configured endpoint | Yes | Yes | Yes |
The only configuration detail required from Application Insights is the connection string. Once you have the connection string, you can configure the agent via your container app's ARM template, with Azure CLI commands or Terraform.
The connection string contains an instrumentation key, which is a unique identifier used to associate telemetry to a specific Application Insights resource. Instrumentation keys aren't security tokens or security keys, and aren't considered secrets.
If you want to protect your Application Insights resource from misuse, see Microsoft Entra authentication for Application Insights. However, the Application Insights resource must allow local authentication to receive data from the OpenTelemetry data agent.
Before you deploy this template, replace the <PLACEHOLDERS>
with your values.
{
...
"properties": {
"appInsightsConfiguration ": {
"connectionString": "<APP_INSIGHTS_CONNECTION_STRING>"
}
"openTelemetryConfiguration": {
...
"tracesConfiguration":{
"destinations": ["appInsights"]
},
"logsConfiguration": {
"destinations": ["appInsights"]
}
}
}
}
The Datadog agent configuration requires a value for site
and key
from your Datadog instance. Gather these values from your Datadog instance according to this table:
Datadog agent property | Container Apps configuration property |
---|---|
DD_SITE |
site |
DD_API_KEY |
key |
Once you have these configuration details, you can configure the agent via your container app's ARM or Bicep template or with Azure CLI commands.
Avoid specifying the value of a secret, such as your Datadog API key, directly in a production environment. Instead, use a reference to a secret stored in Azure Key Vault.
You must enable the key vault for template deployment. To do this, create the key vault with the enabledForTemplateDeployment
property enabled, or run the following Azure CLI command, replacing the <KEY_VAULT_NAME>
with your value:
az keyvault update --name <KEY_VAULT_NAME> --enabled-for-template-deployment true
For more information, see:
Create a parameter file to retrieve your Datadog API key from an Azure Key Vault.
Before you deploy the following files, replace the <PLACEHOLDERS>
with your values.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"datadogapikey": {
"reference": {
"keyVault": {
"id": "/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.KeyVault/vaults/<KEY_VAULT_NAME>"
},
"secretName": "<KEY_VAULT_SECRET_NAME>"
}
}
}
}
You can now reference the datadogapikey
parameter in your ARM template.
{
...
"parameters": {
"datadogapikey": {
"type": "securestring"
}
},
"properties": {
...
"openTelemetryConfiguration": {
...
"destinationsConfiguration":{
...
"dataDogConfiguration":{
"site": "<YOUR_DATADOG_SUBDOMAIN>.datadoghq.com",
"key": "<YOUR_DATADOG_KEY>"
}
},
"tracesConfiguration":{
"destinations": ["dataDog"]
},
"metricsConfiguration": {
"destinations": ["dataDog"]
}
}
}
}
To deploy the resource, run the following Azure CLI command, replacing the <PLACEHOLDERS>
with your values.
az deployment group create \
--resource-group <RESOURCE_GROUP> \
--template-file <ARM_TEMPLATE_FILE> \
--parameters <PARAMETER_FILE>
An OpenTelemetry protocol (OTLP) endpoint is a telemetry data destination that consumes OpenTelemetry data. In your application configuration, you can add multiple OTLP endpoints. The following example adds two endpoints and sends the following data to these endpoints.
Endpoint name | Data sent to endpoint |
---|---|
oltp1 |
Metrics and/or traces |
oltp2 |
Logs and/or traces |
While you can set up as many OTLP-configured endpoints as you like, each endpoint must have a distinct name.
{
"properties": {
"appInsightsConfiguration": {},
"openTelemetryConfiguration": {
"destinationsConfiguration":{
"otlpConfigurations": [
{
"name": "otlp1",
"endpoint": "ENDPOINT_URL_1",
"insecure": false,
"headers": "api-key-1=key"
},
{
"name": "otlp2",
"endpoint": "ENDPOINT_URL_2",
"insecure": true
}
]
},
"logsConfiguration": {
"destinations": ["otlp2"]
},
"tracesConfiguration":{
"destinations": ["otlp1", "otlp2"]
},
"metricsConfiguration": {
"destinations": ["otlp1"]
}
}
}
}
Name | Description |
---|---|
resource-group |
Name of resource group. You can configure the default group using az configure --defaults group=<NAME> . |
name |
Name of the Container Apps environment. |
otlp-name |
A name you select to identify your OTLP-configured endpoint. |
endpoint |
The URL of the destination that receives collected data. |
insecure |
Default true. Defines whether to enable client transport security for the exporter's gRPC connection. If false, the headers parameter is required. |
headers |
Space-separated values, in 'key=value' format, that provides required information for the OTLP endpoints' security. Example: "api-key=key other-config-value=value" . |
To configure an agent, use the destinations
array to define which agents your application sends data. Valid keys are either appInsights
, dataDog
, or the name of your custom OTLP endpoint. You can control how an agent behaves based off data type and endpoint-related options.
Option | Example |
---|---|
Select a data type. | You can configure logs, metrics, and/or traces individually. |
Enable or disable any data type. | You can choose to send only traces and no other data. |
Send one data type to multiple endpoints. | You can send logs to both DataDog and an OTLP-configured endpoint. |
Send different data types to different locations. | You can send traces to an OTLP endpoint and metrics to DataDog. |
Disable sending all data types. | You can choose to not send any data through the OpenTelemetry agent. |
The following example ARM template shows how to use an OTLP endpoint named customDashboard
. It sends:
customDashboard
customDashboard
customDashboard
{
...
"properties": {
...
"openTelemetryConfiguration": {
...
"tracesConfiguration": {
"destinations": [
"appInsights",
"customDashboard"
]
},
"logsConfiguration": {
"destinations": [
"appInsights",
"customDashboard"
]
},
"metricsConfiguration": {
"destinations": [
"dataDog",
"customDashboard"
]
}
}
}
}
The following example template shows how you might configure your container app to collect telemetry data using Azure Monitor Application Insights, Datadog, and with a custom OTLP agent named customDashboard
.
This example works with the parameter file used to retrieve the Datadog API key from an Azure Key Vault.
Before you deploy this template, replace the <PLACEHOLDERS>
with your values.
{
"location": "eastus",
"properties": {
"appInsightsConfiguration": {
"connectionString": "<APP_INSIGHTS_CONNECTION_STRING>"
},
"openTelemetryConfiguration": {
"destinationsConfiguration": {
"dataDogConfiguration": {
"site": "datadoghq.com",
"key": "parameters('datadogapikey')]"
},
"otlpConfigurations": [
{
"name": "customDashboard",
"endpoint": "<OTLP_ENDPOINT_URL>",
"insecure": true
}
]
},
"tracesConfiguration": {
"destinations": [
"appInsights",
"customDashboard"
]
},
"logsConfiguration": {
"destinations": [
"appInsights",
"customDashboard"
]
},
"metricsConfiguration": {
"destinations": [
"dataDog",
"customDashboard"
]
}
}
}
}
For more information, see Microsoft.App/managedEnvironments.
The OpenTelemetry agent automatically injects a set of environment variables into your application at runtime.
The first two environment variables follow standard OpenTelemetry exporter configuration and are used in OTLP standard software development kits. If you explicitly set the environment variable in the container app specification, your value overwrites the automatically injected value.
Learn about the OTLP exporter configuration see, OTLP Exporter Configuration.
Name | Description |
---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
A base endpoint URL for any signal type, with an optionally specified port number. This setting is helpful when you’re sending more than one signal to the same endpoint and want one environment variable to control the endpoint. Example: http://otel.service.k8se-apps:4317/ |
OTEL_EXPORTER_OTLP_PROTOCOL |
Specifies the OTLP transport protocol used for all telemetry data. The managed agent only supports grpc . Value: grpc . |
The other three environment variables are specific to Azure Container Apps, and are always injected. These variables hold agent’s endpoint URLs for each specific data type (logs, metrics, traces).
These variables are only necessary if you're using both the managed OpenTelemetry agent and another OpenTelemetry agent. Using these variables gives you control over how to route data between the different OpenTelemetry agents.
Name | Description | Example |
---|---|---|
CONTAINERAPP_OTEL_TRACING_GRPC_ENDPOINT |
Endpoint URL for trace data only. | http://otel.service.k8se-apps:43178/v1/traces/ |
CONTAINERAPP_OTEL_LOGGING_GRPC_ENDPOINT |
Endpoint URL for log data only. | http://otel.service.k8se-apps:43178/v1/logs/ |
CONTAINERAPP_OTEL_METRIC_GRPC_ENDPOINT |
Endpoint URL for metric data only. | http://otel.service.k8se-apps:43178/v1/metrics/ |
You're billed for the underlying compute of the agent.
See the destination service for their billing structure and terms. For example, if you send data to both Azure Monitor Application Insights and Datadog, you're responsible for the charges applied by both services.
Events
Mar 17, 9 PM - Mar 21, 10 AM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register nowTraining
Module
Implement observability in a .NET cloud-native application with OpenTelemetry - Training
Learn about observability and how to implement it in a cloud-native application. Use OpenTelemetry packages to output logs, metrics, and tracing data and analyze the data in Application Insights and third-party applications.