Edit

Logs Ingestion API in Azure Monitor

  • Article

The Logs Ingestion API in Azure Monitor lets you send data to a Log Analytics workspace using either a REST API call or client libraries. By using this API, you can send data to supported Azure tables or to custom tables that you create. You can even extend the schema of Azure tables with custom columns to accept additional data.

Basic operation

Your application sends data to a data collection endpoint (DCE), which is a unique connection point for your subscription. The payload of your API call includes the source data formatted in JSON. The call:

  • Specifies a data collection rule (DCR) that understands the format of the source data.
  • Potentially filters and transforms the data for the target table.
  • Directs the data to a specific table in a specific workspace.

You can modify the target table and workspace by modifying the DCR without any change to the API call or source data.

Diagram that shows an overview of logs ingestion API.

Note

To migrate solutions from the Data Collector API, see Migrate from Data Collector API and custom fields-enabled tables to DCR-based custom logs.

Components

The Log ingestion API requires the following components to be created before you can send data. Each of these components must all be located in the same region.

Component Description
Data collection endpoint (DCE) The DCE provides an endpoint for the application to send to. A single DCE can support multiple DCRs.
Data collection rule (DCR) Data collection rules define data collected by Azure Monitor and specify how and where that data should be sent or stored. The API call must specify a DCR to use. The DCR must understand the structure of the input data and the structure of the target table. If the two don't match, it can include a transformation to convert the source data to match the target table. You can also use the transformation to filter source data and perform any other calculations or conversions.
Log Analytics workspace The Log Analytics workspace contains the tables that will receive the data. The target tables are specific in the DCR. See Support tables for the tables that the ingestion API can send to.

Supported tables

The following tables can receive data from the ingestion API.

Tables Description
Custom tables The Logs Ingestion API can send data to any custom table that you create in your Log Analytics workspace. The target table must exist before you can send data to it. Custom tables must have the _CL suffix.
Azure tables The Logs Ingestion API can send data to the following Azure tables. Other tables may be added to this list as support for them is implemented.

- CommonSecurityLog
- SecurityEvents
- Syslog
- WindowsEvents

Note

Column names must start with a letter and can consist of up to 45 alphanumeric characters and the characters _ and -. The following are reserved column names: Type, TenantId, resource, resourceid, resourcename, resourcetype, subscriptionid, tenanted. Custom columns you add to an Azure table must have the suffix _CF.

Authentication

Authentication for the Logs Ingestion API is performed at the DCE, which uses standard Azure Resource Manager authentication. A common strategy is to use an application ID and application key as described in Tutorial: Add ingestion-time transformation to Azure Monitor Logs.

Source data

The source data sent by your application is formatted in JSON and must match the structure expected by the DCR. It doesn't necessarily need to match the structure of the target table because the DCR can include a transformation to convert the data to match the table's structure.

Client libraries

You can use the following client libraries to send data to the Logs ingestion API.

REST API call

To send data to Azure Monitor with a REST API call, make a POST call to the DCE over HTTP. Details of the call are described in the following sections.

Endpoint URI

The endpoint URI uses the following format, where the Data Collection Endpoint and DCR Immutable ID identify the DCE and DCR. Stream Name refers to the stream in the DCR that should handle the custom data.

{Data Collection Endpoint URI}/dataCollectionRules/{DCR Immutable ID}/streams/{Stream Name}?api-version=2021-11-01-preview

Note

You can retrieve the immutable ID from the JSON view of the DCR. For more information, see Collect information from the DCR.

Headers

Header Required? Value Description
Authorization Yes Bearer (bearer token obtained through the client credentials flow)
Content-Type Yes application/json
Content-Encoding No gzip Use the gzip compression scheme for performance optimization.
x-ms-client-request-id No String-formatted GUID Request ID that can be used by Microsoft for any troubleshooting purposes.

Body

The body of the call includes the custom data to be sent to Azure Monitor. The shape of the data must be a JSON object or array with a structure that matches the format expected by the stream in the DCR. Additionally, it is important to ensure that the request body is properly encoded in UTF-8 to prevent any issues with data transmission.

Limits and restrictions

For limits related to the Logs Ingestion API, see Azure Monitor service limits.

Next steps