AKS Node Auto Provisioning - NAP

Question

Hello. We have an app deployed on AKS that is pre scaled on our peak business days. It would scale to hundreds of nodes on peak business days. There are multiple node pools for different services. Since NAP is GA, we decide to test it out. NAP is not an option in our prod since pre scaling is a process we follow to be on safe side. The objective of our test was find if NAP could be beneficial both cost and scaling wise on lower environments and other smaller apps. We did a normal test to test the scaling time and NAP was relatively scaling out and scaling in faster than Cluster Autoscaler. But we got significantly high number of errors in NAP compared to CA. If anyone is using NAP technique for scaling, can you explain the benefits, on what workloads are you using this, is there any benefits that could standout than CA etc.

Answer 1

Hello Subin,

Apologies for the late response.

Thanks for the quick follow-up and for sharing the exact error, general-purpose.yaml (NodePool), and aksnodeclass.yaml (AKSNodeClass). This gives us everything we need to pinpoint the root cause.

Quick Summary of the Issue

Your NAP cluster scales out and in faster than the CA cluster (as designed), but the continuous load test with NeoLoad is reporting ~20× more NL-NETWORK-01 (Network IO error sending request) failures. These are not NAP provisioning failures—they are client-side symptoms of temporary pod or endpoint unavailability.

Root cause: Your NodePool uses an aggressive consolidation policy (WhenEmptyOrUnderutilized + consolidateAfter: 30s + 30 % disruption budget). NAP consolidates under-utilized nodes much more quickly than Cluster Autoscaler, which triggers more frequent node drains/evictions. During a high-traffic load test, this leads to brief interruptions in your application endpoints.

Link : https://learn.microsoft.com/en-us/azure/aks/node-auto-provisioning-disruption

This page explains exactly how consolidation, budgets, and consolidateAfter work and why aggressive settings can increase pod disruptions on bursty or load-tested workloads.

Steps to workout :

Make consolidation less aggressive (biggest impact) Update your NodePool with these changes:

• Switch to consolidationPolicy: WhenEmpty (only consolidate truly empty nodes).
• Increase consolidateAfter to at least 5m (or longer for production-like tests).
• Tighten the budget to 10 % (or use a time-based schedule to block disruptions during test windows).

Example updated snippet (replace the disruption section):

disruption:
  budgets:
  - nodes: 10%
  consolidateAfter: 5m
  consolidationPolicy: WhenEmpty

After applying (kubectl apply -f general-purpose.yaml), monitor with:

kubectl get events --field-selector reason=Disruption,reason=Consolidation --sort-by=.metadata.creationTimestamp

Protect your application with a PodDisruptionBudget (PDB) Add a PDB to your sample app Deployment so NAP cannot evict too many pods at once. Example:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-service-pdb
spec:
  minAvailable: 80%      # or a fixed number, e.g. 5
  selector:
    matchLabels:
      app: web-service     # your app's label

1. Apply it before re-running the test. (Note: NAP respects PDBs for most voluntary disruptions.)
2. Quick validation steps
   • Run kubectl get nodes -l karpenter.sh/nodepool=general-purpose -o wide during the test to watch node churn.
   • Check Karpenter controller logs for consolidation decisions:

          kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter --tail=100 | grep -i "consolidat\|disrupt"
     

   • If you still see network errors after the above, share the output of the events command + any pod logs from your app pods—we’ll dig deeper immediately.
3. Optional – Broaden your instance-type options Your current requirements list only allows a handful of small D-series VMs. This works, but for hundreds of nodes it can create more disruption points. Consider relaxing to the full D-family (karpenter.azure.com/sku-family: D) unless you have a specific reason to pin exact SKUs.

These changes keep NAP’s speed and cost benefits while dramatically reducing the error rate you saw versus CA. Many customers run exactly this pattern: NAP in lower environments + tuned disruption settings for production safety.

Apply the NodePool + PDB updates, re-run your load test, and let me know the new error count (or paste any new events/logs). We’ll iterate until it’s rock-solid for your peak-day scaling needs.

Thanks,
Manish.

Answer 2

Node auto-provisioning (NAP) in AKS is designed to complement or replace cluster autoscaler (CA) in scenarios where more flexible, workload‑driven node provisioning is needed.

Key benefits of NAP vs. Cluster Autoscaler

1. More optimal VM selection

• NAP uses pending pod resource requirements (CPU, memory, GPU, etc.) to pick the “best fit” VM SKU and quantity at runtime, instead of only scaling existing node pools.
• Through NodePool requirements, NAP can:
  • Target specific SKU families (D, F, E, L, N) via karpenter.azure.com/sku-family.
  • Target exact SKUs via karpenter.azure.com/sku-name.
  • Control generations via karpenter.azure.com/sku-version.
  • Express constraints like CPU, memory, GPU, accelerated networking, premium storage, etc.
• This can reduce over‑provisioning compared to CA, which typically scales a fixed set of node pool SKUs.

2. Faster, more granular scaling

• NAP provisions, scales, and manages VMs in response to pending pod pressure, similar to Karpenter behavior.
• In practice this often results in:
  • Faster scale‑out when new workloads appear (because NAP can create new nodes tailored to the pods instead of only adding replicas of existing pools).
  • Faster scale‑in at the node level when workloads drain.
• Your observation that NAP scaled faster than CA is consistent with its design.

3. Flexible node pool policies per workload

• NAP introduces Karpenter NodePool and AKSNodeClass CRDs:
  • NodePool defines provisioning policies and constraints for workloads (SKU selectors, zones, capacity type, etc.).
  • AKSNodeClass defines Azure‑specific settings (image, OS disk size, max pods per node, kubelet config, subnet, etc.).
• Multiple NodePools can point to the same AKSNodeClass, so common Azure configuration can be shared while still tailoring scheduling and SKU rules per workload.
• NAP works best when NodePools are mutually exclusive; if multiple pools match, NAP uses the one with the highest weight.

4. Better cost control options

• With NodePool requirements, NAP can:
  • Prefer Spot or On‑demand via karpenter.sh/capacity-type.
  • Constrain max CPU/memory per node and per pool, which helps cap spend.
• Because NAP chooses SKUs based on pod requirements, it can reduce the number of oversized nodes that CA might keep around.

5. Networking and topology flexibility

• AKSNodeClass lets NAP place nodes into specific subnets via vnetSubnetID, while others use the cluster’s default subnet.
• NodePool requirements can target specific availability zones via topology.kubernetes.io/zone.
• This allows:
  • Different subnets for different workloads (e.g., isolation or different routing/security).
  • Multi‑AZ strategies per workload.

Typical workloads where NAP stands out

Based on the documented behavior, NAP tends to be most beneficial for:

1. Highly heterogeneous workloads

• Many different services with different CPU/memory/GPU/storage needs.
• Workloads that would otherwise require many static node pools with carefully chosen SKUs.
• NAP simplifies this by letting workloads drive node shape selection.

2. Event‑driven or bursty workloads

• When combined with KEDA (for pod scaling), NAP can:
  • Scale pods based on events (KEDA) and
  • Provision nodes dynamically for those pods (NAP).
• This is useful for workloads with unpredictable spikes where pre‑scaling is either too expensive or too complex.

3. GPU or specialized hardware workloads

• NAP can target GPU SKUs and capabilities via selectors like karpenter.azure.com/sku-gpu-name, sku-gpu-count, and sku-gpu-manufacturer.
• This is more flexible than maintaining dedicated GPU node pools sized for peak.

4. Multi‑tenant or mixed priority environments

• Different NodePools can:
  • Use different capacity types (Spot vs On‑demand).
  • Use different SKUs and zones.
  • Apply taints/tolerations to isolate workloads.
• This allows cost‑optimized pools for lower‑priority workloads and more reliable pools for critical services.

About the higher error rate you observed

The provided documentation focuses on how to troubleshoot NAP itself rather than comparing error rates with CA. It highlights that:

• NAP is a managed add‑on based on Karpenter.
• Misconfiguration of infrastructure autoscaler components or NodePool/AKSNodeClass can cause issues.
• There is a dedicated troubleshooting guide for NAP that should be followed when errors occur.

If NAP shows more errors than CA in a given environment, the documented next step is to:

• Use the NAP troubleshooting guidance to identify misconfigurations or unsupported scenarios.
• Verify networking configuration (CNI mode, subnet, unsupported features like Calico network policy or dynamic IP allocation).
• Ensure at least one valid NodePool is configured and that taints/tolerations and requirements align with pod specs.

How NAP fits with pre‑scaling strategies

For environments where pre‑scaling to hundreds of nodes on peak days is a strict requirement, CA plus manual/pre‑scheduled scaling may still be preferred for production until NAP behavior is fully validated.

However, for lower environments and smaller apps, NAP can:

• Reduce the need for manual pre‑scaling.
• Optimize VM selection and cost.
• Provide faster, more flexible scaling behavior.

In such cases, NAP is most valuable when workloads are:

• Variable or bursty.
• Diverse in resource requirements.
• Able to tolerate dynamic node shapes and placements defined via NodePool and AKSNodeClass.

---

References:

• Overview of node auto-provisioning (NAP) in Azure Kubernetes Service (AKS)
• Configure node pools for node auto-provisioning (NAP) in Azure Kubernetes Service (AKS)
• Configure AKSNodeClass resources for node auto-provisioning (NAP) in Azure Kubernetes Service (AKS)
• Overview of networking configurations for node auto-provisioning (NAP) in Azure Kubernetes Service (AKS)
• Scaling options for applications in Azure Kubernetes Service (AKS)
• Troubleshoot node auto-provisioning (NAP) in Azure Kubernetes Service (AKS)