Pod placement

Question

I want to schedule pods on regular nodes till they are full and only then make use of virtual nodes. Is that possible?

Answer 1

Hello George Aggrey

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

You can achieve this by using node selectors and taints.

You can use node selectors to specify which nodes your pods should be scheduled on. For example, you can label your regular nodes with a specific label, and then use that label as a node selector in your pod spec. This will ensure that your pods are only scheduled on regular nodes.

Once your regular nodes are full, you can use taints to mark them as unschedulable. This will prevent new pods from being scheduled on those nodes. At the same time, you can add a toleration to your pod spec that allows it to be scheduled on virtual nodes. This will ensure that your pods are scheduled on virtual nodes only when regular nodes are full.

To use virtual nodes in AKS, you need to create a virtual node pool and configure your AKS cluster to use it. You can then use the nodeSelector and tolerations fields in your pod spec to control where your pods are scheduled.

Use taints and tolerations: you could apply a taint to the virtual nodes so nothing will be scheduled there without a specific toleration. This will help make it so the virtual nodes are not used unless there is no where else to schedule the pods.

Configure Node affinity: In the pod spec configure the pods to have an affinity for the nodes labeled as normal nodes.

Tolerations: Add tolerations to the pod spec that allow the pod to be schedule on a virtual node if there are no other nodes to schedule the pod on.

Refer these documents for more details on this:

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
https://learn.microsoft.com/en-us/azure/aks/use-node-taints
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
https://learn.microsoft.com/en-us/azure/aks/operator-best-practices-advanced-scheduler

Hope this helps.
If the suggested response helped you resolve your issue, please 'Accept as answer', so that it can help others in the community looking for help on similar topics.