Data collection rule creation is failing when arm template is deployed

Question

I have an arm template where I am deploying custom tables, data collection endpoint and data collection rule.

Data collection rule fails with the following reason: BadRequest.

Whereas, data collection endpoint and custom tables get deployed as expected.

Resource payload is missing or invalid. (Code: InvalidProperty)

Here, is the arm template, notice that properties of DCR are mentioned,

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "loganalyticsworkspace": {
            "defaultValue": "",
            "type": "String"
        },
        "loganalyticsworkspace-location": {
            "defaultValue": "",
            "type": "String"
        },
        "loganalyticsworkspace-subscription": {
            "defaultValue": "subscription",
            "type": "String",
            "metadata": {
                "description": "Enter the subscription id of log analytics workspace"
            }
        },
        "loganalyticsworkspace-resourceGroup": {
            "defaultValue": "rg_group_name",
            "type": "String",
            "metadata": {
                "description": "Enter the name of the resource group linked with log analytics workspace"
            }
        },
        "Data collection endpoint": {
            "defaultValue": "",
            "type": "String",
            "metadata": {
                "description": "Enter the name of the data collection endpoint that is created"
            }
        },
        "Data collection rule": {
            "defaultValue": "",
            "type": "String",
            "metadata": {
                "description": "Enter the name of the data collection rule that is created"
            }
        }
    },
    "variables": {
        "loganalyticsworkspace": "[parameters('loganalyticsworkspace')]",
        "loganalyticsworkspace-location": "[parameters('loganalyticsworkspace-location')]",
        "loganalyticsworkspace-subscription": "[parameters('loganalyticsworkspace-subscription')]",
        "loganalyticsworkspace-resourceGroup": "[parameters('loganalyticsworkspace-resourceGroup')]",
        "Data collection endpoint": "[parameters('Data collection endpoint')]",
        "Data collection rule": "[parameters('Data collection rule')]",
        "custom-table-auditable-events": "Illumio_Auditable_Events_CL",
        "custom-table-flow-events": "Illumio_Flow_Events_CL"
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionEndpoints",
            "apiVersion": "2022-06-01",
            "name": "[variables('Data collection endpoint')]",
            "location": "[variables('loganalyticsworkspace-location')]",
            "dependsOn": [
                "[resourceId(variables('loganalyticsworkspace-subscription'), variables('loganalyticsworkspace-resourceGroup'), 'Microsoft.Resources/deployments', 'IllumioTablesTemplate')]"
            ],
            "properties": {
                "networkAcls": {
                    "publicNetworkAccess": "Enabled"
                }
            }
        },
        {
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2021-04-01",
            "name": "IllumioTablesTemplate",
            "properties": {
                "mode": "Incremental",
                "template": {
                    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
                    "contentVersion": "1.0.0.0",
                    "parameters": {},
                    "variables": {},
                    "resources": [
                        {
                            "name": "[concat(variables('loganalyticsworkspace'),'/',variables('custom-table-auditable-events'))]",
                            "type": "Microsoft.OperationalInsights/workspaces/tables",
                            "apiVersion": "2022-10-01",
                            "tags": {},
                            "properties": {
                                "plan": "Analytics",
                                "schema": {
                                    "name": "[variables('custom-table-auditable-events')]",
                                    "columns": [
                                        {
                                            "name": "TimeGenerated",
                                            "type": "datetime",
                                            "isDefaultDisplay": true,
                                            "description": "The timestamp (UTC) reflecting the time in which the event was generated."
                                        },
                                        {
                                            "name": "href",
                                            "type": "string"
                                        },
                                        {
                                            "name": "timestamp",
                                            "type": "datetime"
                                        },
                                        {
                                            "name": "pce_fqdn",
                                            "type": "string"
                                        },
                                        {
                                            "name": "created_by",
                                            "type": "dynamic"
                                        },
                                        {
                                            "name": "event_type",
                                            "type": "string"
                                        },
                                        {
                                            "name": "status",
                                            "type": "string"
                                        },
                                        {
                                            "name": "severity",
                                            "type": "string"
                                        },
                                        {
                                            "name": "action",
                                            "type": "dynamic"
                                        },
                                        {
                                            "name": "resource_changes",
                                            "type": "dynamic"
                                        },
                                        {
                                            "name": "notifications",
                                            "type": "dynamic"
                                        },
                                        {
                                            "name": "version",
                                            "type": "int"
                                        }
                                    ]
                                }
                            }
                        },
                        {
                            "name": "[concat(variables('loganalyticsworkspace'),'/',variables('custom-table-flow-events'))]",
                            "type": "Microsoft.OperationalInsights/workspaces/tables",
                            "apiVersion": "2022-10-01",
                            "tags": {},
                            "properties": {
                                "plan": "Analytics",
                                "schema": {
                                    "name": "[variables('custom-table-flow-events')]",
                                    "columns": [
                                        {
                                            "name": "TimeGenerated",
                                            "type": "datetime",
                                            "isDefaultDisplay": true,
                                            "description": "The timestamp (UTC) reflecting the time in which the event was generated."
                                        },
                                        {
                                            "name": "pn",
                                            "type": "string"
                                        },
                                        {
                                            "name": "un",
                                            "type": "string"
                                        },
                                        {
                                            "name": "src_ip",
                                            "type": "string"
                                        },
                                        {
                                            "name": "dst_ip",
                                            "type": "string"
                                        },
                                        {
                                            "name": "class",
                                            "type": "string"
                                        },
                                        {
                                            "name": "proto",
                                            "type": "int"
                                        },
                                        {
                                            "name": "dst_port",
                                            "type": "int"
                                        },
                                        {
                                            "name": "flow_count",
                                            "type": "int"
                                        },
                                        {
                                            "name": "dir",
                                            "type": "string"
                                        },
                                        {
                                            "name": "timestamp",
                                            "type": "datetime"
                                        },
                                        {
                                            "name": "state",
                                            "type": "string"
                                        },
                                        {
                                            "name": "pd_qualifier",
                                            "type": "int"
                                        },
                                        {
                                            "name": "pd",
                                            "type": "int"
                                        },
                                        {
                                            "name": "src_hostname",
                                            "type": "string"
                                        },
                                        {
                                            "name": "src_href",
                                            "type": "string"
                                        },
                                        {
                                            "name": "dst_hostname",
                                            "type": "string"
                                        },
                                        {
                                            "name": "dst_href",
                                            "type": "string"
                                        },
                                        {
                                            "name": "network",
                                            "type": "string"
                                        },
                                        {
                                            "name": "src_labels",
                                            "type": "dynamic"
                                        },
                                        {
                                            "name": "dst_labels",
                                            "type": "dynamic"
                                        },
                                        {
                                            "name": "interval_sec",
                                            "type": "int"
                                        },
                                        {
                                            "name": "pce_fqdn",
                                            "type": "string"
                                        },
                                        {
                                            "name": "version",
                                            "type": "int"
                                        }
                                    ]
                                }
                            }
                        }
                    ]
                },
                "parameters": {}
            },
            "subscriptionId": "[variables('loganalyticsworkspace-subscription')]",
            "resourceGroup": "[variables('loganalyticsworkspace-resourceGroup')]"
        },
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "apiVersion": "2022-06-01",
            "name": "[variables('Data collection rule')]",
            "location": "[variables('loganalyticsworkspace-location')]",
            "dependsOn": [
                "[resourceId(variables('loganalyticsworkspace-subscription'), variables('loganalyticsworkspace-resourceGroup'), 'Microsoft.Insights/dataCollectionEndpoints', variables('Data collection endpoint'))]"
            ],
            "tags": {
                "createdBy": "Sentinel"
            },
            "properties": {
                "dataCollectionEndpointId": "[resourceId(variables('loganalyticsworkspace-subscription'), variables('loganalyticsworkspace-resourceGroup'), 'Microsoft.Insights/dataCollectionEndpoints', variables('Data collection endpoint'))]",
                "streamDeclarations": {
                    "[concat('Custom-',variables('custom-table-auditable-events'))]": {
                        "columns": [
                            {
                                "name": "href",
                                "type": "string"
                            },
                            {
                                "name": "timestamp",
                                "type": "datetime"
                            },
                            {
                                "name": "pce_fqdn",
                                "type": "string"
                            },
                            {
                                "name": "created_by",
                                "type": "dynamic"
                            },
                            {
                                "name": "event_type",
                                "type": "string"
                            },
                            {
                                "name": "status",
                                "type": "string"
                            },
                            {
                                "name": "severity",
                                "type": "string"
                            },
                            {
                                "name": "action",
                                "type": "dynamic"
                            },
                            {
                                "name": "resource_changes",
                                "type": "dynamic"
                            },
                            {
                                "name": "notifications",
                                "type": "dynamic"
                            },
                            {
                                "name": "version",
                                "type": "int"
                            }
                        ]
                    },
                    "[concat('Custom-',variables('custom-table-flow-events'))]": {
                        "columns": [
                            {
                                "name": "pn",
                                "type": "string"
                            },
                            {
                                "name": "un",
                                "type": "string"
                            },
                            {
                                "name": "src_ip",
                                "type": "string"
                            },
                            {
                                "name": "dst_ip",
                                "type": "string"
                            },
                            {
                                "name": "class",
                                "type": "string"
                            },
                            {
                                "name": "proto",
                                "type": "int"
                            },
                            {
                                "name": "dst_port",
                                "type": "int"
                            },
                            {
                                "name": "count",
                                "type": "int"
                            },
                            {
                                "name": "dir",
                                "type": "string"
                            },
                            {
                                "name": "timestamp",
                                "type": "datetime"
                            },
                            {
                                "name": "state",
                                "type": "string"
                            },
                            {
                                "name": "pd_qualifier",
                                "type": "int"
                            },
                            {
                                "name": "pd",
                                "type": "int"
                            },
                            {
                                "name": "src_hostname",
                                "type": "string"
                            },
                            {
                                "name": "src_href",
                                "type": "string"
                            },
                            {
                                "name": "dst_hostname",
                                "type": "string"
                            },
                            {
                                "name": "dst_href",
                                "type": "string"
                            },
                            {
                                "name": "network",
                                "type": "string"
                            },
                            {
                                "name": "src_labels",
                                "type": "dynamic"
                            },
                            {
                                "name": "dst_labels",
                                "type": "dynamic"
                            },
                            {
                                "name": "interval_sec",
                                "type": "int"
                            },
                            {
                                "name": "pce_fqdn",
                                "type": "string"
                            },
                            {
                                "name": "version",
                                "type": "int"
                            }
                        ]
                    }
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "name": "[variables('loganalyticsworkspace')]",
                            "workspaceResourceId": "[resourceId(variables('loganalyticsworkspace-subscription'), variables('loganalyticsworkspace-resourceGroup'), 'Microsoft.OperationalInsights/Workspaces', variables('loganalyticsworkspace'))]"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "[concat('Custom-',variables('custom-table-flow-events'))]"
                        ],
                        "destinations": "[variables('loganalyticsworkspace')]",
                        "transformKql": "source\n| extend TimeGenerated = todatetime(timestamp)\n\n",
                        "outputStream": "[concat('Custom-',variables('custom-table-auditable-events'))]"
                    },
                    {
                        "streams": [
                            "[concat('Custom-',variables('custom-table-flow-events'))]"
                        ],
                        "destinations": "[variables('loganalyticsworkspace')]",
                        "transformKql": "source\n| extend TimeGenerated = todatetime(timestamp)\n| project-rename flow_count=['count']\n\n",
                        "outputStream": "[concat('Custom-',variables('custom-table-flow-events'))]"
                    }
                ]
            }
        }
    ],
    "outputs": {}
}

Answer 1

@Ashwin Venkatesha - Welcome to Microsoft Q&A and thanks for reaching out to us.

AFAIK, the error usually indicates that there is an issue with the properties that you are providing for the data collection rule.

Based on the ARM template you provided, the issue may be related to the "streamDeclarations" property, and it looks like the column names in the "streamDeclarations" property do not match the column names in the custom tables that you are creating.

Please ensure that the column names in the "streamDeclarations" property match the column names in the custom tables. And recheck if the "event_type" column is not missing in the auditable events.

Once you have made the necessary changes to the ARM template, you can try deploying it again to see if the issue has been resolved.

Hope this helps. and please feel free to reach out if you have any further questions.

---

Please don't forget to "Accept as Answer" and click "Yes" if the above response is helpful, so it can be beneficial to the community.