AKS Taints & Tolerations

AzureUser-9588 151 Reputation points
2024-06-14T08:16:57.4766667+00:00

How to ensure AKS system pods to run only on System node pool and application pods to run only on the user node pools?

I do NOT want application pods to be on system node pool and system pods to be on user node pool.

I see that creating a dedicated system node pool and applying the CriticalAddonsOnly=true:NoSchedule taint can prevent application pods from being scheduled on system node pools. However, didn't see on how to avoid running system pods on user node pool.

Having system pods in user node pool is causing autoscaling to malfunction while scaling IN.

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,987 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Adharsh Santhanam 2,705 Reputation points
    2024-06-14T16:44:18.2133333+00:00

    Hello AzureUser-9588, as you rightly pointed out, creating a dedicated system node pool with the taint will ensure that application pods (without the tolerations) are prevented from getting scheduled in the system node pool. Similarly, to ensure that system pods are not getting scheduled on user node pools, add some specific taint to the user node pool and ensure that your application pods alone have tolerations towards that specific taint. This will ensure that the system pods aren't getting scheduled on the user node pools. Also, leverage node affinity and try setting it to requiredDuringSchedulingIgnoredDuringExecution.

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments

  2. Anveshreddy Nimmala 3,460 Reputation points Microsoft Vendor
    2024-06-18T05:46:14.0433333+00:00

    Hello AzureUser-9588

    Taints don't directly affect system pods operating on user nodes, but they do stop application pods from scheduling on system nodes.

    For better resource allocation, think about utilizing a different VM size or SKU for the system node pool than for the user node pools.

    For redundancy, establish a different system node pool with a minimum of two nodes. This pool will be used exclusively to run metrics-server and kube-dns, two essential system pods.

    Use the CriticalAddonsOnly=true property.NoSchedule contaminates the pool of system nodes.

    Make changes to your deployments (such as kube-dns) that manage system pods, and include a tolerance that corresponds to the taint on the system node pool.

    apiVersion: apps/v1

    kind: Deployment
    metadata:
      name: demo-deployment
    spec:
      template:
        spec:
          tolerations:
          - effect: NoSchedule
            operator: Exists
            selector:
              matchLabels:
                kubernetes.io/system-node: ""
    

    With this setup, application pods will be routed to the user and system pods will be scheduled on the system node pool with tolerance.

    Hope this helps you

    0 comments No comments