Data collection transformations in Azure Monitor
Transformations in Azure Monitor allow you to filter or modify incoming data before it's sent to a Log Analytics workspace. This article provides a basic description of transformations and how they are implemented. It provides links to other content for actually creating a transformation.
Why to use transformations
The following table describes the different goals that transformations can be used to achieve.
|Remove sensitive data||You may have a data source that sends information you don't want stored for privacy or compliancy reasons.
Filter sensitive information. Filter out entire rows or just particular columns that contain sensitive information.
Obfuscate sensitive information. For example, you might replace digits with a common character in an IP address or telephone number.
|Enrich data with additional or calculated information||Use a transformation to add information to data that provides business context or simplifies querying the data later.
Add a column with additional information. For example, you might add a column identifying whether an IP address in another column is internal or external.
Add business specific information. For example, you might add a column indicating a company division based on location information in other columns.
|Reduce data costs||Since you're charged ingestion cost for any data sent to a Log Analytics workspace, you want to filter out any data that you don't require to reduce your costs.
Remove entire rows. For example, you might have a diagnostic setting to collect resource logs from a particular resource but not require all of the log entries that it generates. Create a transformation that filters out records that match a certain criteria.
Remove a column from each row. For example, your data may include columns with data that's redundant or has minimal value. Create a transformation that filters out columns that aren't required.
Parse important data from a column. You may have a table with valuable data buried in a particular column. Use a transformation to parse the valuable data into a new column and remove the original.
Transformations may be applied to the following tables in a Log Analytics workspace.
- Any Azure table listed in Tables that support transformations in Azure Monitor Logs
- Any custom table
How transformations work
Transformations are performed in Azure Monitor in the data ingestion pipeline after the data source delivers the data and before it's sent to the destination. The data source may perform its own filtering before sending data but then rely on the transformation for further manipulation for it's sent to the destination.
Transformations are defined in a data collection rule (DCR) and use a Kusto Query Language (KQL) statement that is applied individually to each entry in the incoming data. It must understand the format of the incoming data and create output in the structure expected by the destination.
For example, a DCR that collects data from a virtual machine using Azure Monitor agent would specify particular data to collect from the client operating system. It could also include a transformation that would get applied to that data after it's sent to the data ingestion pipeline that further filters the data or adds a calculated column. This workflow is shown in the following diagram.
Another example is data sent from a custom application using the logs ingestion API. In this case, the application sends the data to a data collection endpoint and specifies a data collection rule in the REST API call. The DCR includes the transformation and the destination workspace and table.
Workspace transformation DCR
The workspace transformation DCR is a special DCR that's applied directly to a Log Analytics workspace. It includes default transformations for one more supported tables. These transformations are applied to any data sent to these tables unless that data came from another DCR.
For example, if you create a transformation in the workspace transformation DCR for the
Event table, it would be applied to events collected by virtual machines running the Log Analytics agent since this agent doesn't use a DCR. The transformation would be ignored by any data sent from the Azure Monitor agent though since it uses a DCR and would be expected to provide its own transformation.
Creating a transformation
There are multiple methods to create transformations depending on the data collection method. The following table lists guidance for different methods for creating transformations.
|Logs ingestion API with transformation||Send data to Azure Monitor Logs using REST API (Azure portal)
Send data to Azure Monitor Logs using REST API (Resource Manager templates)
|Transformation in workspace DCR||Add workspace transformation to Azure Monitor Logs using the Azure portal
Add workspace transformation to Azure Monitor Logs using resource manager templates
Cost for transformations
There is no direct cost for transformations, but you may incur charges for the following:
- If your transformation increases the size of the incoming data, adding a calculated column for example, then you're charged at the normal rate for ingestion of that additional data.
- If your transformation reduces the incoming data by more than 50%, then you're charged for ingestion of the amount of filtered data above 50%.
The formula to determine the filter ingestion charge from transformations is
[GB filtered out by transformations] - ( [Total GB ingested] / 2 ). For example, suppose that you ingest 100 GB on a particular day, and transformations remove 70 GB. You would be charged for 70 GB - (100 GB / 2) or 20 GB. To avoid this charge, you should use other methods to filter incoming data before the transformation is applied.
See Azure Monitor pricing for current charges for ingestion and retention of log data in Azure Monitor.
If Azure Sentinel is enabled for the Log Analytics workspace, then there is no filtering ingestion charge regardless of how much data the transformation filters.
- Create a data collection rule and an association to it from a virtual machine using the Azure Monitor agent.