Enable control plane logs in Azure Red Hat OpenShift with hosted control planes (preview)

You can forward control plane logs from an Azure Red Hat OpenShift with hosted control planes cluster to an Azure destination by configuring Azure Monitor diagnostic settings. You select which control plane log categories to collect and where to store or stream them. Azure Monitor diagnostic settings connect to the Microsoft.RedHatOpenShift/HCPOpenShiftClusters resource type to collect these logs.

Prerequisites

  • An Azure Red Hat OpenShift with hosted control planes cluster is deployed and running.
  • Azure CLI version 2.67.0 or later is installed. Run az --version to check your installed version. To install or upgrade, see Install the Azure CLI.
  • You have sufficient Azure subscription permissions to create storage accounts or Event Hubs namespaces and to create diagnostic settings on the Azure Red Hat OpenShift with hosted control planes cluster resource. At minimum, you need Monitoring Contributor on the cluster resource, plus Storage Account Contributor for storage account destinations or Azure Event Hubs Data Owner for Event Hub destinations.

Select control plane log categories

Select the log categories to include in your diagnostic setting. You don't need to enable all categories. Select only the categories relevant to your monitoring, troubleshooting, or compliance requirements.

Category Description
kube-apiserver Kubernetes API server logs. Records requests processed by the API server, including authentication, authorization, and admission decisions.
kube-audit Kubernetes API audit logs for all requests. Captures every API request, including reads, writes, and admin operations. This category generates a high volume of log data.
kube-audit-admin Kubernetes API audit logs for admin and write requests only. This category excludes read-only requests, which reduces log volume compared to kube-audit.
kube-controller-manager Kubernetes controller manager logs. Records activity from controllers that manage cluster state, such as replication, endpoints, and namespace lifecycle.
kube-scheduler Kubernetes scheduler logs. Records scheduling decisions, including pod placement, resource constraints, and node selection.
cloud-controller-manager Azure cloud controller manager logs. Records interactions between Kubernetes and Azure cloud resources, such as load balancers, routes, and node lifecycle.
cluster-autoscaler Cluster autoscaler logs. Records decisions about adding and removing nodes based on pending pods and underutilized nodes.
capi-provider Cluster API (CAPI) provider logs. Records activity from the Cluster API provider that manages nodes, networks, and disks for the cluster.
csi-azuredisk-controller Azure Disk Container Storage Interface (CSI) driver controller logs. Records volume provisioning, attachment, and lifecycle operations for Azure managed disks.
csi-azurefile-controller Azure File CSI driver controller logs. Records volume provisioning and lifecycle operations for Azure file shares.
csi-snapshot-controller CSI volume snapshot controller logs. Records volume snapshot creation, deletion, and restore operations.

For the full list of supported log categories, see Supported logs for Microsoft.RedHatOpenShift/hcpOpenShiftClusters.

Note

The kube-audit and kube-audit-admin categories overlap. Enabling both captures a complete audit trail alongside a filtered view for operational monitoring, but duplicates write and admin request entries. Consider whether the added storage cost is justified for your compliance needs.

Forward control plane logs to a storage account

You can forward control plane logs to an Azure storage account for long-term storage and analysis.

Set environment variables for storage account forwarding

Set the following environment variables. Replace each placeholder value with your own values.

SUBSCRIPTION=$(az account show --query id --output tsv)
CUSTOMER_RG_NAME="<resource-group-name>"
LOCATION="<location>"
CLUSTER_NAME="<cluster-name>"
STORAGE_ACCOUNT_NAME="<storage-account-name>"
DIAGNOSTIC_SETTING_NAME="<diagnostic-setting-name>"

Create the storage account

If you don't already have a storage account, create one. The storage account doesn't need to be in the same resource group as your cluster, but it must be in the same subscription. If you use an existing storage account in a different resource group, adjust the --storage-account resource path in the diagnostic setting command accordingly.

az storage account create \
  --name ${STORAGE_ACCOUNT_NAME} \
  --resource-group ${CUSTOMER_RG_NAME} \
  --location ${LOCATION} \
  --sku Standard_LRS \
  --kind StorageV2

Create the diagnostic setting for storage

Create a diagnostic setting to collect control plane logs and send them to the storage account. The following example collects all available log categories. For descriptions of each category, see Select control plane log categories.

az monitor diagnostic-settings create \
    --name ${DIAGNOSTIC_SETTING_NAME} \
    --resource /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/${CLUSTER_NAME} \
    --storage-account /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.Storage/storageAccounts/${STORAGE_ACCOUNT_NAME} \
    --logs '[
      {"category": "kube-apiserver",            "enabled": true},
      {"category": "kube-audit",                "enabled": true},
      {"category": "kube-audit-admin",          "enabled": true},
      {"category": "kube-controller-manager",   "enabled": true},
      {"category": "kube-scheduler",            "enabled": true},
      {"category": "cloud-controller-manager",  "enabled": true},
      {"category": "cluster-autoscaler",        "enabled": true},
      {"category": "capi-provider",             "enabled": true},
      {"category": "csi-azuredisk-controller",  "enabled": true},
      {"category": "csi-azurefile-controller",  "enabled": true},
      {"category": "csi-snapshot-controller",   "enabled": true}
    ]'

To collect only specific categories, set "enabled": false for the categories you want to exclude.

Verify the diagnostic setting for storage

Verify that you created the diagnostic setting successfully.

az monitor diagnostic-settings list \
    --resource /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/${CLUSTER_NAME} \
    --query "[?name=='${DIAGNOSTIC_SETTING_NAME}']"

The output includes an entry with the name you specified, showing the log categories with enabled set to true and the correct storage account resource ID as the destination.

Note

Logs typically appear in your storage account within 35 to 40 minutes. Each log category creates a container with an insights- prefix (for example, insights-logs-kube-apiserver).

Forward control plane logs to an event hub

You can forward control plane logs to an Azure Event Hub for real-time streaming and integration with downstream systems.

Set environment variables for event hub forwarding

Set the following environment variables. Replace each placeholder value with your own values.

SUBSCRIPTION=$(az account show --query id --output tsv)
CUSTOMER_RG_NAME="<resource-group-name>"
LOCATION="<location>"
CLUSTER_NAME="<cluster-name>"
NAMESPACE_NAME="<event-hubs-namespace-name>"
EVENT_HUB_NAME="<event-hub-name>"
AUTH_RULE_NAME="<authorization-rule-name>"
DIAGNOSTIC_SETTING_NAME="<diagnostic-setting-name>"

Create the Event Hubs namespace

Create an Event Hubs namespace. The namespace must use the Standard tier or higher and must be in the same subscription as the cluster.

az eventhubs namespace create \
    --name ${NAMESPACE_NAME} \
    --resource-group ${CUSTOMER_RG_NAME} \
    --location ${LOCATION} \
    --sku Standard

Create the event hub

Create an event hub within the namespace.

az eventhubs eventhub create \
    --name ${EVENT_HUB_NAME} \
    --namespace-name ${NAMESPACE_NAME} \
    --resource-group ${CUSTOMER_RG_NAME}

Create the authorization rule

Create an authorization rule with Send rights on the namespace. The diagnostic setting uses this rule to send log data to the event hub.

az eventhubs namespace authorization-rule create \
    --resource-group ${CUSTOMER_RG_NAME} \
    --namespace-name ${NAMESPACE_NAME} \
    --name ${AUTH_RULE_NAME} \
    --rights Send

Create the diagnostic setting for event hub

Create a diagnostic setting to forward control plane logs to the event hub. The following example forwards all available log categories. For descriptions of each category, see Select control plane log categories.

az monitor diagnostic-settings create \
    --name ${DIAGNOSTIC_SETTING_NAME} \
    --resource /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/${CLUSTER_NAME} \
    --event-hub ${EVENT_HUB_NAME} \
    --event-hub-rule /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.EventHub/namespaces/${NAMESPACE_NAME}/authorizationRules/${AUTH_RULE_NAME} \
    --logs '[
      {"category": "kube-apiserver",            "enabled": true},
      {"category": "kube-audit",                "enabled": true},
      {"category": "kube-audit-admin",          "enabled": true},
      {"category": "kube-controller-manager",   "enabled": true},
      {"category": "kube-scheduler",            "enabled": true},
      {"category": "cloud-controller-manager",  "enabled": true},
      {"category": "cluster-autoscaler",        "enabled": true},
      {"category": "capi-provider",             "enabled": true},
      {"category": "csi-azuredisk-controller",  "enabled": true},
      {"category": "csi-azurefile-controller",  "enabled": true},
      {"category": "csi-snapshot-controller",   "enabled": true}
    ]'

To forward only specific categories, set "enabled": false for the categories you want to exclude.

Verify the diagnostic setting for event hub

Verify that you created the diagnostic setting successfully.

az monitor diagnostic-settings list \
    --resource /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/${CLUSTER_NAME} \
    --query "[?name=='${DIAGNOSTIC_SETTING_NAME}']"

The output includes an entry with the name you specified, showing the log categories with enabled set to true and the correct event hub authorization rule as the destination.

Note

Logs typically appear in the event hub within 45 minutes. You can verify delivery by going to your event hub in the Azure portal and checking the Data Explorer, or by consuming messages with the Azure Event Hubs SDK.

Clean up resources

If you created resources for testing and you no longer need them, delete the diagnostic setting and the destination resources to stop incurring costs.

Delete a diagnostic setting:

az monitor diagnostic-settings delete \
    --name ${DIAGNOSTIC_SETTING_NAME} \
    --resource /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/${CLUSTER_NAME}

Delete a storage account:

az storage account delete \
    --name ${STORAGE_ACCOUNT_NAME} \
    --resource-group ${CUSTOMER_RG_NAME} \
    --yes

Delete an Event Hubs namespace (this action also deletes all event hubs and authorization rules within the namespace):

az eventhubs namespace delete \
    --name ${NAMESPACE_NAME} \
    --resource-group ${CUSTOMER_RG_NAME}

Related content

  • Last updated on