Data Collection Rule ARM template fails when using a custom table

Question

Data Collection Rule ARM template fails when using a custom table

Scoot-3223 91

Looking for help resolving an issue with this tutorial: Send data to Azure Monitor using Logs ingestion API, via ARM. The Portal version of the tutorial works fine but the ARM version fails when deploying the ARM template for the Data Collection Rule.

Analysis reveals the Output Stream of the DCR must start with the word Custom- or an exception is thrown. So the DCR template is good, but, the custom table that's created via the CLI in the ARM tutorial cannot have the Custom- prefix, as table names cannot have special characters.

So I cannot find a way to create a custom table that's acceptable for the Data Collection Rule.

This works fine in the Portal as when the Data Collection Rule is created, the table is created automatically behind the scenes.

I appreciate any help!

ARM deployment error when using the specified template.

error": {
        "code": "InvalidPayload",
        "message": "Data collection rule is invalid",
        "details": [
            {
                "code": "InvalidOutputTable",
                "target": "properties.dataFlows[0]",
                "message": "Table for output stream 'Custom-MyTable_CL' is not available for destination 'myworkspace'."
            }

Scoot-3223 91 Reputation points

2023-08-15T18:32:07.38+00:00
Since tables cannot be created like, "Custom-MyTable_CL", the tutorial specifies creating the table with name "MyTable_CL". So if you change the Output Stream to "MyTable_CL", you get this deployment error:

"code": "InvalidStream", "target": "Properties.DataFlows[0].OutputStream", "message": "'Output Stream' stream 'MyTable_CL' must be a custom stream or one of: Microsoft-AACAudit,Microsoft-AACHttpRequest,
SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2023-08-23T10:43:03.6533333+00:00

Thanks for your question. I have tested the scenario from the document you have mentioned and was able to deploy the Data Collection Rule using ARM Template.

Can you please share how you have created the custom table along with naming convention and as well as the ARM template?

Also as you mentioned in your comment, kindly specify the table as MyTable_CL format and don't change the outputstream format. Try providing the naming convention as Custom-MyTable_CL for output stream.
Scoot-3223 91 Reputation points

2023-08-24T21:14:20.8+00:00

The solution to my problem was figuring out how to manually create the custom table. It cannot be created in the portal as that forces you to create it in Log Analytics which simultaneously creates the DCR. Creating the table via the azapi provider resource works great. You can then use the arm template to create the DCR. So in all, I'm able to use Terraform to fully automate the creation of the endpoint, the custom table, the data collection rule. With this done I have a function app that writes logs to the custom table in real time.

2 answers

Your answer

Scoot-3223 91 Reputation points

2023-08-15T18:32:07.38+00:00

Since tables cannot be created like, "Custom-MyTable_CL", the tutorial specifies creating the table with name "MyTable_CL". So if you change the Output Stream to "MyTable_CL", you get this deployment error:

"code": "InvalidStream", "target": "Properties.DataFlows[0].OutputStream", "message": "'Output Stream' stream 'MyTable_CL' must be a custom stream or one of: Microsoft-AACAudit,Microsoft-AACHttpRequest,
SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2023-08-23T10:43:03.6533333+00:00

Thanks for your question. I have tested the scenario from the document you have mentioned and was able to deploy the Data Collection Rule using ARM Template.

Can you please share how you have created the custom table along with naming convention and as well as the ARM template?

Also as you mentioned in your comment, kindly specify the table as MyTable_CL format and don't change the outputstream format. Try providing the naming convention as Custom-MyTable_CL for output stream.
Scoot-3223 91 Reputation points

2023-08-24T21:14:20.8+00:00

The solution to my problem was figuring out how to manually create the custom table. It cannot be created in the portal as that forces you to create it in Log Analytics which simultaneously creates the DCR. Creating the table via the azapi provider resource works great. You can then use the arm template to create the DCR. So in all, I'm able to use Terraform to fully automate the creation of the endpoint, the custom table, the data collection rule. With this done I have a function app that writes logs to the custom table in real time.

Answer 1

Scoot-3223 91

Posted my answer in the most recent comment above. Hopefully this can be of some use.

Answer 2

@Scoot-3223 Thanks for sharing your solution here. It would be beneficial to the customers who are facing similar issue.

As per community rules, one can't accept their own answer. So, summarizing the answer provided by you in answer section for community benefit.

The resolution to customer issue involved understanding the process of manually generating the custom table. The custom table cannot be generated through the portal, as it enforces the creation within Log Analytics, simultaneously generating the Data Collection Rule (DCR). Creating the table through the azapi provider resource proves to be effective. Subsequently, the arm template can be utilized to produce the DCR. In summary, customer has fully automated the endpoint, custom table, and data collection rule creation using Terraform. Consequently, they have achieved a functional app that writes real-time logs to the custom table.

Share via

Data Collection Rule ARM template fails when using a custom table

2 answers

Your answer