Edit

Platform log collection with data collection rules (Preview)

Platform logs are logs generated by Azure platform services and resources. Platform telemetry data collection rules (DCRs) let you send platform logs from supported Azure resources to a Log Analytics workspace, storage account, or Event Hubs. This article shows you how to create a DCR that exports platform logs.

Note

Platform logs in Azure Monitor are also referred to as resource logs.

Compare to diagnostic settings

Before this feature, you could only export platform logs by using diagnostic settings. Diagnostic settings are still required for resource types that don't yet support DCRs.

Platform telemetry DCRs provide several benefits over diagnostic settings:

  • DCRs are more flexible and scalable than diagnostic settings. Use a single DCR with multiple resources, while a separate diagnostic setting is required for each resource.
  • Use consistent ARM, Bicep, and Terraform deployment patterns across resource types.
  • Use managed identity for keyless authentication to storage accounts and Event Hubs.

Important

This feature is currently in preview. See Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

Before you create a platform telemetry DCR, make sure that you have the following:

Tool-specific prerequisites

Tool Requirement
Azure portal No extra tooling required.
Azure CLI Azure CLI 2.61 or later.
Azure PowerShell Az.Monitor 5.2 or later.
REST API Bearer token for ARM authentication.
Bicep or ARM templates Azure CLI or Azure PowerShell for deployment.

Export destinations

A platform telemetry DCR supports one destination type. To send data to multiple destination types, create separate DCRs.

Destination type Region requirement Managed identity Details
Log Analytics workspace DCR and workspace must be in the same region. Not required Data is stored in resource-specific log tables.
Storage account DCR, storage account, and monitored resources must be in the same region. Required (Storage Blob Data Contributor) Data is written as JSON blobs in a container.
Event Hubs DCR, Event Hubs namespace, and monitored resources must be in the same region. Required (Azure Event Hubs Data Sender) Data is streamed as JSON events.

Note

It may take up to 30 minutes for logs to show up in the destination after the initial setup, and for the data to appear in the destination.

Create a data collection rule

Use one of the following methods to create a platform telemetry DCR to collect platform logs. After you create the DCR, grant permissions to the managed identity for storage account and Event Hubs destinations, and then create a data collection rule association.

Note

For storage account and Event Hubs destinations, the DCR, destination, and monitored resources must all be in the same region.

Create a data collection rule using the Azure portal

  1. On the Monitor menu in the Azure portal, select Data Collection Rules and then Create.
  2. On the Create Data Collection Rule page, enter a rule name, select a Subscription, Resource group, and Region for the DCR.
  3. Select PlatformTelemetry for the Type of telemetry and Enable Managed Identity if you want to send logs to a Storage Account or Event Hubs. A screenshot showing the basics tab of the create data collection rule page.
  4. On the Resources page, select Add resources to add the resources you want to collect logs from.
  5. Select Next to move to the Collect and deliver tab. A screenshot showing the resources tab of the create data collection rule page.
  6. Select Add new datasource.
  7. The resource type of the resource specified in the previous step is automatically selected. Add more resource types if you want to use this rule to collect logs from multiple resource types in the future. Select the Actions for a resource type if you want to remove some of the logs collected for it. By default, all available logs for the resource are collected. A screenshot showing the Data source tab of the Add new data source pane for a data collection rule.
  8. Select Next Destinations to move to the Destinations tab.
  9. Select Add destination and then the Destination type that you want to add. The required fields change based on the destination type you select. A screenshot showing the Destination tab of the Add new data source pane for a data collection rule.
  10. Select Save , then select Review + create.

Grant permissions to the managed identity

For storage account and Event Hubs destinations, the DCR system-assigned managed identity must have write permissions on the destination resource.

Destination type Required role Role definition ID
Log Analytics workspace Not required N/A
Storage account Storage Blob Data Contributor ba92f5b4-2d11-453d-a403-e96b0029c9fe
Event Hubs Azure Event Hubs Data Sender 2b629674-e913-4c01-ae53-ef4638d8f975
# Storage account
az role assignment create \
  --assignee "<dcr-principal-id>" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>"

# Event Hubs
az role assignment create \
  --assignee "<dcr-principal-id>" \
  --role "Azure Event Hubs Data Sender" \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.EventHub/namespaces/<event-hub-namespace>"

Create a data collection rule association

After you create the DCR and assign permissions, associate the rule with each resource you want to monitor.

az monitor data-collection rule association create \
  --name "<association-name>" \
  --rule-id "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Insights/dataCollectionRules/<dcr-name>" \
  --resource "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/<resource-provider>/<resource-type>/<resource-name>"

Verify data collection

After you create the DCR and association, allow up to 30 minutes for the first data to appear. Once data starts flowing, latency is approximately three minutes.

The following tabs show how to verify data collection for each destination type.

Query the resource-specific log table in your Log Analytics workspace. For example, to verify logs from Azure Database for MySQL flexible servers:

AzureDiagnostics
| where TimeGenerated > ago(1h)
| where ResourceProvider == "MICROSOFT.DBFORMYSQL"
| summarize count() by Category
| order by count_ desc

You can also run a broad search to locate any data for a resource in the last 60 minutes:

search "<resource-name-or-id-fragment>"
| where TimeGenerated > ago(60m)
| project TimeGenerated, Type, _ResourceId, Resource, ResourceGroup
| take 50

Troubleshoot

If you don't see data flowing, use DCR monitoring features in Azure Monitor:

  • Check Logs Ingestion Bytes per Min and Logs Rows Received per Min to confirm that data is reaching Azure Monitor.
  • Check Logs Rows Dropped per Min and Logs Transformation Errors per Min for processing errors.
  • Enable DCR error logs and query the DCRLogErrors table for detailed error information.

Common issues to check:

  • Incorrect API version: Use api-version=2024-03-11 for DCR and DCRA operations.
  • Streams mismatch with resource type. Confirm that the stream specification matches a supported resource type and log category.
  • Region mismatch. Ensure the DCR, destination resource, and monitored resources are all in the same region (except for Log Analytics workspace destinations).
  • Missing managed identity role assignment: For storage account and Event Hubs destinations, verify that Storage Blob Data Contributor or Azure Event Hubs Data Sender is assigned.

Limitations

  • Only one destination type can be specified per DCR. To send data to multiple destination types, create separate DCRs.
  • A maximum of five platform telemetry DCRs can be associated with a single Azure resource.
  • While you can use DCRs and diagnostic settings simultaneously, disable diagnostic settings for logs when you use DCRs to avoid duplicate data collection.
  • The DCR and destination resource (workspace, storage account, or event hub namespace) must be in the same Azure region. Create a separate DCR per region and destination as needed.
  • For storage account and Event Hubs destinations, monitored resources must also be in the same region as the DCR and destination.