Custom data collection rule not applying transformKQL

Question

As per the article https://learn.microsoft.com/en-us/cli/azure/monitor/data-collection/rule?view=azure-cli-latest#az-monitor-data-collection-rule-update I'm trying to update my Data Collection Rule with a transformKql query in the dataFlows section of the configuration but getting an error.

• Are we not allowed to update default rule?
• I am thinking replacing the stream group (Microsoft-ContainerInsights-Group-Default) with the individual stream I want to modify, which for container logs is Microsoft-ContainerInsights-ContainerLogV2?

PS C:\Users\12345678> az monitor data-collection rule update --name MSCI-eastus2-az-aks-na-abc-d01 --resource-group rg-nonprod-abc-xyz-d01 --set "dataFlows[0].transformKql='source | project-away TenantId, LogSource, Type, _ResourceId'"

az : ERROR: (InvalidPayload) Data collection rule is invalid

At line:1 char:1

• az monitor data-collection rule update --name MSCI-eastus2-az-aks-na-ab ...

• 
      + CategoryInfo          : NotSpecified: (ERROR: (Invalid...rule is invalid:String) [], RemoteException
  
      + FullyQualifiedErrorId : NativeCommandError
  
   
  
  

Code: InvalidPayload

Message: Data collection rule is invalid

Exception Details: (InvalidDataFlow) Data flows containing stream groups cannot have custom transformation.

Code: InvalidDataFlow

Message: Data flows containing stream groups cannot have custom transformation.

Target: Properties.DataFlows[0].TransformKql

Answer 1

When updating a Data Collection Rule (DCR) in Azure Monitor, you cannot apply a custom transformation using transformKql if your data flow contains stream groups. The error message you received indicates that the data flow you are trying to update includes a stream group, which is not compatible with custom transformations.

To resolve this issue, you would need to either:

1. Remove the stream group from the data flow if you want to apply a custom transformation.
2. Alternatively, if you need to keep the stream group, you will not be able to use transformKql in that specific data flow.

Regarding your question about replacing the stream group with an individual stream, you can do that, but ensure that the new configuration does not include any stream groups if you intend to use custom transformations.

Answer 2

Hello Pravin Patel, You tried to update an Azure Monitor Data Collection Rule (DCR) with a transformKql query in the dataFlows section. The error you received was: Code: Data flows containing stream groups cannot have custom transformation. This happened because you attempted to apply a transformation to the default stream group (Microsoft-ContainerInsights-Group-Default).

Many AKS/Container Insights DCRs created by the portal/extension are solution‑managed (effectively read‑only for some sections). You can usually update destinations, but adding transforms to a dataFlow that references a streamGroup isn’t allowed. If the DCR is solution‑managed, the clean path is to clone to a custom DCR and re‑associate it to the AKS cluster.

“Replace the stream group with the individual stream Microsoft-ContainerInsights-ContainerLogV2?” Yes, that is the supported pattern for transformations. Put the exact individual stream(s) you want under streams (not streamGroups) and then add transformKql.

Since you're considering replacing the stream group with Microsoft-ContainerInsights-ContainerLogV2, that is the right approach! You can definitely modify your DCR to specify individual streams instead of using the default group.

To proceed, you might want to:

• Remove the Stream Group: Update your DCR to include the individual streams where you want to apply your transformations.
• Update the KQL Query: Make sure your transformation query is accurately targeting the fields you want to process.

Here's a sample command structure you can try:

az monitor data-collection rule update --name <Your-DCR-Name> --resource-group <Your-Resource-Group> --set "dataFlows[0].streams=['Microsoft-ContainerInsights-ContainerLogV2']" --set "dataFlows[0].transformKql='source | project-away TenantId, LogSource, Type, _ResourceId'"

If you still face any issue, please help us with the below details.

1. Have you already attempted to edit the data flow to include only the individual streams?
2. Are there any specific logs or data types you're particularly interested in transforming?
3. Could you provide the current complete configuration of your DCR related to the data flows?

Reference document:

Data collection rules in Azure Monitor - Azure Monitor | Microsoft Learn

Create a transformation in Azure Monitor - Azure Monitor | Microsoft Learn

Transformations Azure Monitor - Azure Monitor | Microsoft Learn

Monitor DCR data collection in Azure Monitor - Azure Monitor | Microsoft Learn

Hope this helps. Thanks.