Transform or customize data at ingestion time in Microsoft Sentinel (preview)

This article describes how to configure ingestion-time data transformation and custom log ingestion for use in Microsoft Sentinel.

Ingestion-time data transformation provides customers with more control over the ingested data. Supplementing the pre-configured, hardcoded workflows that create standardized tables, ingestion time-transformation adds the capability to filter and enrich the output tables, even before running any queries. Custom log ingestion uses the Custom Log API to normalize custom-format logs so they can be ingested into certain standard tables, or alternatively, to create customized output tables with user-defined schemas for ingesting these custom logs.

These two mechanisms are configured using Data Collection Rules (DCRs), either in the Log Analytics portal, or via API or ARM template. This article will help you choose which kind of DCR you need for your particular data connector, and direct you to the instructions for each scenario.

Prerequisites

Before you start configuring DCRs for data transformation:

• Learn more about data transformation and DCRs in Azure Monitor and Microsoft Sentinel. For more information, see:

  • Data collection rules in Azure Monitor
  • Logs ingestion API in Azure Monitor Logs (Preview)
  • Transformations in Azure Monitor Logs (preview)
  • Data transformation in Microsoft Sentinel (preview)

• Verify data connector support. Make sure that your data connectors are supported for data transformation.

  In our data connector reference article, check the section for your data connector to understand which types of DCRs are supported. Continue in this article to understand how the DCR type you select affects the rest of the ingestion and transformation process.

Determine your requirements

If you are ingesting	Ingestion-time transformation is...	Use this DCR type
Custom data through the Log Ingestion API	Required Included in the DCR that defines the data model	Standard DCR
Built-in data types (Syslog, CommonSecurityLog, WindowsEvent, SecurityEvent) using the Azure Monitor Agent	Optional If desired, added to the DCR that configures how this data is being ingested	Standard DCR
Built-in data types from most other sources	Optional If desired, added to the DCR attached to the Workspace where this data is being ingested	Workspace transformation DCR

Configure your data transformation

Use the following procedures from the Log Analytics and Azure Monitor documentation to configure your data transformation DCRs:

Direct ingestion through the Log Ingestion API:

• Walk through a tutorial for ingesting logs using the Azure portal.
• Walk through a tutorial for ingesting logs using Azure Resource Manager (ARM) templates and REST API.

Workspace transformations:

• Walk through a tutorial for configuring workspace transformation using the Azure portal.
• Walk through a tutorial for configuring workspace transformation using Azure Resource Manager (ARM) templates and REST API.-

More on data collection rules:

• Structure of a data collection rule in Azure Monitor (preview)
• Data collection transformations in Azure Monitor (preview)

When you're done, come back to Microsoft Sentinel to verify that your data is being ingested based on your newly configured transformation. It may take up to 60 minutes for the data transformation configurations to apply.

Migrate to ingestion-time data transformation

If you currently have custom Microsoft Sentinel data connectors, or built-in, API-based data connectors, you may want to migrate to using ingestion-time data transformation.

Use one of the following methods:

• Configure a DCR to define, from scratch, the custom ingestion from your data source to a new table. You might use this option if you want to use a new schema that doesn't have the current column suffixes, and doesn't require query-time KQL functions to standardize your data.

  After you've verified that your data is properly ingested to the new table, you can delete the legacy table, as well as your legacy, custom data connector.

• Continue using the custom table created by your custom data connector. You might use this option if you have a lot of custom security content created for your existing table. In such cases, see Migrate from Data Collector API and custom fields-enabled tables to DCR-based custom logs in the Azure Monitor documentation.

Next steps

For more information about data transformation and DCRs, see:

• Custom data ingestion and transformation in Microsoft Sentinel (preview)
• Data collection transformations in Azure Monitor Logs (preview)
• Logs ingestion API in Azure Monitor Logs (Preview)
• Structure of a data collection rule in Azure Monitor (preview)
• Configure data collection for the Azure Monitor Agent