AKS Taints & Tolerations

Question

How to ensure AKS system pods to run only on System node pool and application pods to run only on the user node pools?

I do NOT want application pods to be on system node pool and system pods to be on user node pool.

I see that creating a dedicated system node pool and applying the CriticalAddonsOnly=true:NoSchedule taint can prevent application pods from being scheduled on system node pools. However, didn't see on how to avoid running system pods on user node pool.

Having system pods in user node pool is causing autoscaling to malfunction while scaling IN.

Answer 1

Hello AzureUser-9588, as you rightly pointed out, creating a dedicated system node pool with the taint will ensure that application pods (without the tolerations) are prevented from getting scheduled in the system node pool. Similarly, to ensure that system pods are not getting scheduled on user node pools, add some specific taint to the user node pool and ensure that your application pods alone have tolerations towards that specific taint. This will ensure that the system pods aren't getting scheduled on the user node pools. Also, leverage node affinity and try setting it to requiredDuringSchedulingIgnoredDuringExecution.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hello AzureUser-9588

Taints don't directly affect system pods operating on user nodes, but they do stop application pods from scheduling on system nodes.

For better resource allocation, think about utilizing a different VM size or SKU for the system node pool than for the user node pools.

For redundancy, establish a different system node pool with a minimum of two nodes. This pool will be used exclusively to run metrics-server and kube-dns, two essential system pods.

Use the CriticalAddonsOnly=true property.NoSchedule contaminates the pool of system nodes.

Make changes to your deployments (such as kube-dns) that manage system pods, and include a tolerance that corresponds to the taint on the system node pool.

apiVersion: apps/v1

kind: Deployment
metadata:
  name: demo-deployment
spec:
  template:
    spec:
      tolerations:
      - effect: NoSchedule
        operator: Exists
        selector:
          matchLabels:
            kubernetes.io/system-node: ""

With this setup, application pods will be routed to the user and system pods will be scheduled on the system node pool with tolerance.

Hope this helps you