Share via

Removing Prometheus Metrics from AKS Cluster

Jamie 0 Reputation points
2023-12-20T12:41:37.15+00:00

We run a Kubernetes cluster with a number of Windows Server 2022 nodes, each running about 50 pods. Whilst troubleshooting the cause of an OS disk filling up, we enabled Prometheus Metrics for a production AKS cluster. Although we've now found the root cause of that, new issues have begun since we enabled Prometheus Metrics.

The main issue we have observed is a repeated PFN_REFERENCE_COUNT Blue Screen of Death in TCPIP.SYS. We've also noticed containerd is printing, every 50ms, a "Stats" line to the containerd log file, which seems to be consuming a lot of the container's available IOPS.

I am not completely sure the crashes are directly related, but I would like to disable Prometheus Metrics to rule this out.

However, I can't find a clear-cut documented way to explain how to disable this monitoring.

Azure Kubernetes Service
Azure Kubernetes Service
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,447 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Anveshreddy Nimmala 3,550 Reputation points Microsoft External Staff Moderator
    2023-12-21T05:16:31.6166667+00:00

    Hi Jamie,

    Open the Azure portal and navigate to your AKS cluster.

    1.Click on the "Monitoring" tab in the left-hand menu.

    2.Click on the "Azure Monitor for containers" link.

    3.Click on the "Data Collection" tab.

    4.Click on the "Edit" button.

    5.Toggle the "Enable Prometheus Metrics" switch to the "Off" position.

    6.Click on the "Save" button.

    7.This will disable Prometheus Metrics for your AKS cluster. It may take a few minutes for the changes to take effect.

    If you want to completely remove Prometheus Metrics from your AKS cluster.

    you can follow these additional steps:

    1.Open a terminal window and connect to your AKS cluster using the kubectl command-line tool.

    2.Run the following command to delete the Prometheus Metrics addon:

    3.kubectl delete daemon-set azure monitor-containers-Prometheus -n kube-system

    4.This will delete the Prometheus Metrics addon from your AKS cluster.

    5.Run the following command to delete the Prometheus Metrics service:

    6.kubectl delete service Prometheus -n kube-system

    7.This will delete the Prometheus Metrics service from your AKS cluster.

    8.Removing Prometheus Metrics from your AKS cluster will also remove any data that was collected by the addon.

    please refer the documentation for the same.

    https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-disable

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.