Tutorial: Send data to Azure Monitor using Logs ingestion API (Resource Manager templates)

Článok
10/27/2023

The Logs Ingestion API in Azure Monitor allows you to send custom data to a Log Analytics workspace. This tutorial uses Azure Resource Manager templates (ARM templates) to walk through configuration of the components required to support the API and then provides a sample application using both the REST API and client libraries for .NET, Go, Java, JavaScript, and Python.

Note

This tutorial uses ARM templates to configure the components required to support the Logs ingestion API. See Tutorial: Send data to Azure Monitor Logs with Logs ingestion API (Azure portal) for a similar tutorial that uses the Azure portal UI to configure these components.

The steps required to configure the Logs ingestion API are as follows:

Create a Microsoft Entra application to authenticate against the API.
Create a data collection endpoint (DCE) to receive data.
Create a custom table in a Log Analytics workspace. This is the table you'll be sending data to.
Create a data collection rule (DCR) to direct the data to the target table.
Give the Microsoft Entra application access to the DCR.
See Sample code to send data to Azure Monitor using Logs ingestion API for sample code to send data to using the Logs ingestion API.

Prerequisites

To complete this tutorial, you need:

A Log Analytics workspace where you have at least contributor rights.
Permissions to create DCR objects in the workspace.

Collect workspace details

Start by gathering information that you'll need from your workspace.

Go to your workspace in the Log Analytics workspaces menu in the Azure portal. On the Properties page, copy the Resource ID and save it for later use.

Create Microsoft Entra application

Start by registering a Microsoft Entra application to authenticate against the API. Any Resource Manager authentication scheme is supported, but this tutorial follows the Client Credential Grant Flow scheme.

On the Microsoft Entra ID menu in the Azure portal, select App registrations > New registration.
Give the application a name and change the tenancy scope if the default isn't appropriate for your environment. A Redirect URI isn't required.
After registration, you can view the details of the application. Note the Application (client) ID and the Directory (tenant) ID. You'll need these values later in the process.
Generate an application client secret, which is similar to creating a password to use with a username. Select Certificates & secrets > New client secret. Give the secret a name to identify its purpose and select an Expires duration. The option 12 months is selected here. For a production implementation, you would follow best practices for a secret rotation procedure or use a more secure authentication mode, such as a certificate.
Select Add to save the secret and then note the Value. Ensure that you record this value because you can't recover it after you leave this page. Use the same security measures as you would for safekeeping a password because it's the functional equivalent.

Create data collection endpoint

A DCE is required to accept the data being sent to Azure Monitor. After you configure the DCE and link it to a DCR, you can send data over HTTP from your application. The DCE must be located in the same region as the DCR and the Log Analytics workspace where the data will be sent.

In the Azure portal's search box, enter template and then select Deploy a custom template.
Select Build your own template in the editor.

Paste the following ARM template into the editor and then select Save. You don't need to modify this template because you'll provide values for its parameters.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionEndpointName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Endpoint to create."
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "westus2",
            "metadata": {
                "description": "Specifies the location for the Data Collection Endpoint."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionEndpoints",
            "name": "[parameters('dataCollectionEndpointName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2021-04-01",
            "properties": {
                "networkAcls": {
                "publicNetworkAccess": "Enabled"
                }
            }
        }
    ],
    "outputs": {
        "dataCollectionEndpointId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionEndpoints', parameters('dataCollectionEndpointName'))]"
        }
    }
}

On the Custom deployment screen, specify a Subscription and Resource group to store the DCR and then provide values like a Name for the DCE. The Location should be the same location as the workspace. The Region will already be populated and will be used for the location of the DCE.
Select Review + create and then select Create after you review the details.
Select JSON View to view other details for the DCE. Copy the Resource ID and the logsIngestion endpoint which you'll need in a later step.

Create new table in Log Analytics workspace

The custom table must be created before you can send data to it. The table for this tutorial will include five columns shown in the schema below. The name, type, and description properties are mandatory for each column. The properties isHidden and isDefaultDisplay both default to false if not explicitly specified. Possible data types are string, int, long, real, boolean, dateTime, guid, and dynamic.

Note

This tutorial uses PowerShell from Azure Cloud Shell to make REST API calls by using the Azure Monitor Tables API. You can use any other valid method to make these calls.

Important

Custom tables must use a suffix of _CL.

Select the Cloud Shell button in the Azure portal and ensure the environment is set to PowerShell.

Copy the following PowerShell code and replace the variables in the Path parameter with the appropriate values for your workspace in the Invoke-AzRestMethod command. Paste it into the Cloud Shell prompt to run it.

$tableParams = @'
{
    "properties": {
        "schema": {
            "name": "MyTable_CL",
            "columns": [
                {
                    "name": "TimeGenerated",
                    "type": "datetime",
                    "description": "The time at which the data was generated"
                },
               {
                    "name": "Computer",
                    "type": "string",
                    "description": "The computer that generated the data"
                },
                {
                    "name": "AdditionalContext",
                    "type": "dynamic",
                    "description": "Additional message properties"
                },
                {
                    "name": "CounterName",
                    "type": "string",
                    "description": "Name of the counter"
                },
                {
                    "name": "CounterValue",
                    "type": "real",
                    "description": "Value collected for the counter"
                }
            ]
        }
    }
}
'@

Invoke-AzRestMethod -Path "/subscriptions/{subscription}/resourcegroups/{resourcegroup}/providers/microsoft.operationalinsights/workspaces/{workspace}/tables/MyTable_CL?api-version=2022-10-01" -Method PUT -payload $tableParams

Create data collection rule

The DCR defines how the data will be handled once it's received. This includes:

Schema of data that's being sent to the endpoint
Transformation that will be applied to the data before it's sent to the workspace
Destination workspace and table the transformed data will be sent to

In the Azure portal's search box, enter template and then select Deploy a custom template.
Select Build your own template in the editor.

Paste the following ARM template into the editor and then select Save.

Notice the following details in the DCR defined in this template:

dataCollectionEndpointId: Resource ID of the data collection endpoint.
streamDeclarations: Column definitions of the incoming data.
destinations: Destination workspace.
dataFlows: Matches the stream with the destination workspace and specifies the transformation query and the destination table. The output of the destination query is what will be sent to the destination table.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRuleName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Rule to create."
            }
        },
        "location": {
            "type": "string",
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Rule."
            }
        },
        "workspaceResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Log Analytics workspace to use."
            }
        },
        "endpointResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Data Collection Endpoint to use."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "name": "[parameters('dataCollectionRuleName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2021-09-01-preview",
            "properties": {
                "dataCollectionEndpointId": "[parameters('endpointResourceId')]",
                "streamDeclarations": {
                    "Custom-MyTableRawData": {
                        "columns": [
                            {
                                "name": "Time",
                                "type": "datetime"
                            },
                            {
                                "name": "Computer",
                                "type": "string"
                            },
                            {
                                "name": "AdditionalContext",
                                "type": "string"
                            },
                            {
                                "name": "CounterName",
                                "type": "string"
                            },
                            {
                                "name": "CounterValue",
                                "type": "real"
                            }
                        ]
                    }
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "[parameters('workspaceResourceId')]",
                            "name": "myworkspace"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-MyTableRawData"
                        ],
                        "destinations": [
                            "myworkspace"
                        ],
                        "transformKql": "source | extend jsonContext = parse_json(AdditionalContext) | project TimeGenerated = Time, Computer, AdditionalContext = jsonContext, CounterName=tostring(jsonContext.CounterName), CounterValue=toreal(jsonContext.CounterValue)",
                        "outputStream": "Custom-MyTable_CL"
                    }
                ]
            }
        }
    ],
    "outputs": {
        "dataCollectionRuleId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))]"
        }
    }
}

On the Custom deployment screen, specify a Subscription and Resource group to store the DCR. Then provide values defined in the template. The values include a Name for the DCR and the Workspace Resource ID that you collected in a previous step. The Location should be the same location as the workspace. The Region will already be populated and will be used for the location of the DCR.
Select Review + create and then select Create after you review the details.
When the deployment is complete, expand the Deployment details box and select your DCR to view its details. Select JSON View.
Copy the Immutable ID for the DCR. You'll use it in a later step when you send sample data using the API.

Note

All the properties of the DCR, such as the transformation, might not be displayed in the Azure portal even though the DCR was successfully created with those properties.

Assign permissions to a DCR

After the DCR has been created, the application needs to be given permission to it. Permission will allow any application using the correct application ID and application key to send data to the new DCE and DCR.

From the DCR in the Azure portal, select Access Control (IAM) > Add role assignment.
Select Monitoring Metrics Publisher and select Next. You could instead create a custom action with the Microsoft.Insights/Telemetry/Write data action.
Select User, group, or service principal for Assign access to and choose Select members. Select the application that you created and choose Select.
Select Review + assign and verify the details before you save your role assignment.