Events
Mar 17, 9 PM - Mar 21, 10 AM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register nowThis browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Transformations in Azure Monitor allow you to filter or modify incoming data before it's sent to a Log Analytics workspace. Transformations are performed in the cloud pipeline after the data source delivers the data and before it's sent to the destination. They're defined in a data collection rule (DCR) and use a Kusto Query Language (KQL) statement that's applied individually to each entry in the incoming data.
The following diagram illustrates the transformation process for incoming data and shows a sample query that might be used. In this sample, only records where the message
column contains the word error
will be collected.
The following tables in a Log Analytics workspace support transformations.
There are some data collection scenarios that will allow you to add a transformation using the Azure portal, but most scenarios will require you to create a new DCR using its JSON definition or add a transformation to an existing DCR. See Create a transformation in Azure Monitor for different options and Best practices and samples for transformations in Azure Monitor for sample transformation queries for common scenarios.
Transformations are defined in a data collection rule (DCR), but there are still data collections in Azure Monitor that don't yet use a DCR. Examples include resource logs collected by diagnostic settings and application data collected by Application insights.
The workspace transformation data collection rule (DCR) is a special DCR that's applied directly to a Log Analytics workspace. The purpose of this DCR is to perform transformations on data that does not yet use a DCR for its data collection, and thus has no means to define a transformation.
There can be only one workspace DCR for each workspace, but it can include transformations for any number of supported tables. These transformations are applied to any data sent to these tables unless that data came from another DCR.
For example, the Event table is used to store events from Windows virtual machines. If you create a transformation in the workspace transformation DCR for the Event table, it would be applied to events collected by virtual machines running the Log Analytics agent1 because this agent doesn't use a DCR. The transformation would be ignored though by any data sent from Azure Monitor Agent (AMA) because it uses a DCR to define its data collection. You can still use a transformation with Azure Monitor agent, but you would include that transformation in the DCR associated with the agent and not the workspace transformation DCR.
1 The Log Analytics agent has been deprecated, but some environments may still use it. It's only one example of a data source that doesn't use a DCR.
While transformations themselves don't incur direct costs, the following scenarios can result in additional charges:
To calculate the data processing charge resulting from transformations, use the following formula:
[GB filtered out by transformations] - ([GB data ingested by pipeline] / 2). The following table shows examples.
Data ingested by pipeline | Data dropped by transformation | Data ingested by Log Analytics workspace | Data processing charge | Ingestion charge |
---|---|---|---|---|
20 GB | 12 GB | 8 GB | 2 GB 1 | 8 GB |
20 GB | 8 GB | 12 GB | 0 GB | 12 GB |
1 This charge excludes the charge for data ingested by Log Analytics workspace.
To avoid this charge, you should filter ingested data using alternative methods before applying transformations. By doing so, you can reduce the amount of data processed by transformations and, therefore, minimize any additional costs.
See Azure Monitor pricing for current charges for ingestion and retention of log data in Azure Monitor.
Important
If Azure Sentinel is enabled for the Log Analytics workspace, there's no filtering ingestion charge regardless of how much data the transformation filters.
Events
Mar 17, 9 PM - Mar 21, 10 AM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register nowTraining
Module
Code-free transformation at scale with Azure Data Factory - Training
Perform code-free transformation at scale with Azure Data Factory or Azure Synapse Pipeline
Certification
Microsoft Certified: Azure Data Engineer Associate - Certifications
Demonstrate understanding of common data engineering tasks to implement and manage data engineering workloads on Microsoft Azure, using a number of Azure services.