AKS integration with ACI

han chong tan 21 Reputation points
2022-04-29T08:08:59.74+00:00

I have AKS with advanced network (CNI) setup, I'm interested to enable virtual nodes. I understand that if its enabled, then when there's a need to scale the pods beyond the current limit of a node, it will scale to ACI, without the delay of bringing up another node. My question is does it automatically scales the pods in ACI down to 0, after my node has scaled up to accommodate more pods (I have auto scale enabled for my node). How does it work in this situation?

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,858 questions
0 comments
Accepted answer
  1. shiva patpi 13,141 Reputation points Microsoft Employee
    2022-05-01T06:49:47.71+00:00

    Hello @han chong tan ,
    I was testing your scenario and the answer is NO.
    Here is the reason :
    When you implement Virtual Nodes and if you want your PODS to be scheduled on those Virtual Nodes - you might have to implement below 2 parts in combination with HPA.

    1. Node Affinity
    2. Tolerations

    Based upon the HPA CPU utilization % it will scale up the PODS , when you have a nodeAffinity like below to reduce the CPU utilization based upon the HPA parameters - If the nodes are not not available , PODS will go and sit on ACI - there by CPU utilization % decreases. So even if a new node gets added as there are no pending PODS + CPU utilization % is less than the target % - hence no new PODS and existing PODS on ACI will not come down to 0.

    Example Below YAML:-
    spec:
    affinity:
    nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 1
    preference:
    matchExpressions:
    - key: type
    operator: NotIn
    values:
    - virtual-kubelet
    containers:
    - name: php-apache
    image: k8s.gcr.io/hpa-example
    ports:
    - containerPort: 80
    resources:
    limits:
    cpu: 500m
    requests:
    cpu: 500m
    tolerations:
    - key: virtual-kubelet.io/provider
    operator: Exists
    - effect: NoSchedule
    key: azure.com/aci

    2 people found this answer helpful.
    0 comments

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. han chong tan 21 Reputation points
    2022-05-09T08:05:05.42+00:00

    HI @shiva patpi , sorry for the delay. I have a long break and forgotten to check on this. To clarify on your suggestion/answer, can I say that it means cluster auto scaling (node scaling) can't be used together with virtual node (ACI)? Because when both is enabled, would it just be adding pods into ACI and not bring up a new node (when max pods per node is hit)?

    200203-image.png

    0 comments
  2. han chong tan 21 Reputation points
    2022-05-09T08:16:10.183+00:00

    Hi @shiva patpi , I think I get what you meant. So we can have cluster auto scaling/node scaling enabled, together with virtual node enabled. Then it's up to how we use node affinity & tolerations in the pod yaml configuration to make it scale to the virtual node instead of normal node (non-virtual) when scaling is required. Once its there in the virtual node, it doesn't move it back to a normal node by itself. Is my understanding correct?