Edit

Transform or customize data at ingestion time in Microsoft Sentinel (preview)

  • Article

This article describes how to configure ingestion-time data transformation and custom log ingestion for use in Microsoft Sentinel.

Ingestion-time data transformation provides customers with more control over the ingested data. Supplementing the pre-configured, hardcoded workflows that create standardized tables, ingestion time-transformation adds the capability to filter and enrich the output tables, even before running any queries. Custom log ingestion uses the Custom Log API to normalize custom-format logs so they can be ingested into certain standard tables, or alternatively, to create customized output tables with user-defined schemas for ingesting these custom logs.

These two mechanisms are configured using Data Collection Rules (DCRs), either in the Log Analytics portal, or via API or ARM template. This article will help you choose which kind of DCR you need for your particular data connector, and direct you to the instructions for each scenario.

Prerequisites

Before you start configuring DCRs for data transformation:

Determine your requirements

If you are ingesting Ingestion-time transformation is... Use this DCR type
Custom data through
the Log Ingestion API
  • Required
  • Included in the DCR that defines the data model
    		•  Standard DCR
    Built-in data types
    (Syslog, CommonSecurityLog, WindowsEvent, SecurityEvent)
    using the legacy Log Analytics Agent (MMA)
  • Optional
  • If desired, added to the DCR attached to the Workspace where this data is being ingested
    		•  Workspace transformation DCR
    Built-in data types
    from most other sources
  • Optional
  • If desired, added to the DCR attached to the Workspace where this data is being ingested
    		•  Workspace transformation DCR

    Configure your data transformation

    Use the following procedures from the Log Analytics and Azure Monitor documentation to configure your data transformation DCRs:

    Direct ingestion through the Log Ingestion API:

    Workspace transformations:

    More on data collection rules:

    When you're done, come back to Microsoft Sentinel to verify that your data is being ingested based on your newly configured transformation. It may take up to 60 minutes for the data transformation configurations to apply.

    Migrate to ingestion-time data transformation

    If you currently have custom Microsoft Sentinel data connectors, or built-in, API-based data connectors, you may want to migrate to using ingestion-time data transformation.

    Use one of the following methods:

    • Configure a DCR to define, from scratch, the custom ingestion from your data source to a new table. You might use this option if you want to use a new schema that doesn't have the current column suffixes, and doesn't require query-time KQL functions to standardize your data.

      After you've verified that your data is properly ingested to the new table, you can delete the legacy table, as well as your legacy, custom data connector.

    • Continue using the custom table created by your custom data connector. You might use this option if you have a lot of custom security content created for your existing table. In such cases, see Migrate from Data Collector API and custom fields-enabled tables to DCR-based custom logs in the Azure Monitor documentation.

    Next steps

    For more information about data transformation and DCRs, see: