AKS Node configuration mismatch

Question

AKS Node configuration mismatch

David Poor 0

Summary: AKS system nodepool in persistent Failed provisioning state with corrupted metadata. Nodepool has disappeared from az aks nodepool list but underlying VMs and nodes are still running and healthy.

Symptoms: - Nodepool "system" shows Failed provisioningState - vmSize and count are blank/null in nodepool metadata - Upgrade attempt fails: "QuotaExceeded - unable to create minimum required surge nodes" - After update attempt, nodepool no longer appears in az aks nodepool list - However 2 VMs are still running and healthy. kubectl get nodes shows both nodes Ready - All workload pods are running on these nodes - Nodepool type: VirtualMachines fleet (not VMSS)

Commands run and their output: az aks nodepool list → system nodepool now missing entirely az vm list → both VMs present and Succeeded kubectl get nodes → both nodes Ready v1.32.4 az aks nodepool show --name system --query status → {"provisioningError": null}

Request: Please reconcile the AKS control plane metadata for the system nodepool so that: 1. Nodepool reappears in az aks nodepool list 2. provisioningState returns to Succeeded 3. vmSize and count are correctly populated 4. Upgrades can proceed without data loss

Prasad Chaganti 775 Reputation points Microsoft External Staff Moderator

2026-07-08T18:59:12.9933333+00:00
Hi David Poor,

Thank you for sharing the detailed symptoms and command outputs.

Based on the information provided, this looks like the AKS resource provider/control-plane metadata for the system node pool is not aligned with the underlying node/VM state. Since the underlying VMs are still running and kubectl get nodes shows the nodes as Ready, this appears to be a control-plane/node pool metadata issue rather than an immediate node health issue.

Microsoft documentation recommends validating both the AKS cluster provisioning state and the node pool provisioning state, and also checking the underlying VM/VMSS or VM resources, activity logs, quota/capacity, policies, permissions, and locks when a node pool is stuck in Failed state. The AKS failed-state troubleshooting guidance also notes that reconciliation can be triggered by running an Azure Resource Manager update against the AKS cluster resource.

Please try the following validation/reconciliation steps first:

# Confirm AKS cluster state az aks show --resource-group <resource-group> --name <cluster-name> --query "{provisioningState:provisioningState, powerState:powerState.code, kubernetesVersion:kubernetesVersion}" -o table # Get AKS cluster resource ID AKS_ID=$(az aks show --resource-group <resource-group> --name <cluster-name> --query id -o tsv) # Trigger AKS managed cluster reconciliation az resource update --ids $AKS_ID

After the reconciliation completes, validate whether the node pool is visible again:

az aks nodepool list --resource-group <resource-group> --cluster-name <cluster-name> -o table az aks nodepool show --resource-group <resource-group> --cluster-name <cluster-name> --name system -o json

Also review the Azure Activity Log around the time of the failed upgrade/update operation to identify the exact failure code and operation that left the node pool in the failed state:

az monitor activity-log list --resource-group <resource-group> --status Failed --max-events 50 -o table

Regarding the QuotaExceeded - unable to create minimum required surge nodes message: AKS upgrades might require temporary surge capacity. Microsoft documentation notes that quota/capacity limitations can block create, scale, or upgrade operations, and for allocation/capacity-related upgrade failures, using maxUnavailable instead of surge can help avoid creating additional buffer nodes during upgrade. This should only be tried after the node pool metadata is restored and the node pool is visible again.

Note: Please avoid directly deleting, resizing, or modifying the underlying VMs outside the AKS API. AKS manages the lifecycle of agent nodes, and direct customisation of the underlying IaaS resources is not supported because such changes may not persist through AKS update, scale, upgrade, or reboot operations.

If the system node pool still does not appear in az aks nodepool list after cluster reconciliation, this will likely require Microsoft Support/AKS backend investigation because the customer-side AKS API no longer has a valid node pool object to operate on.

Referenced documents:

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/availability-performance/cluster-node-virtual-machine-failed-state

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/create-upgrade-delete/aks-at-scale-troubleshoot-guide

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/availability-performance/node-not-ready-basic-troubleshooting

Please review above shared steps and let us know if you have any further queries.
Bamidele 0 Reputation points

2026-07-08T22:27:18.95+00:00

Hello @Prasad Chaganti thanks for the breakdown, I have gone through the process, and still reconciliation fails after steps indidcated, and nodepool still doesn't show. Myself and @David Poor were trying to create a support request for this issue, but were forced to only access this Q&A. How do we raise the ticket, cause we think we are at that point of backend intervention as we speak.

Your answer

Bamidele 0 Reputation points

2026-07-08T22:27:18.95+00:00

Hello @Prasad Chaganti thanks for the breakdown, I have gone through the process, and still reconciliation fails after steps indidcated, and nodepool still doesn't show. Myself and @David Poor were trying to create a support request for this issue, but were forced to only access this Q&A. How do we raise the ticket, cause we think we are at that point of backend intervention as we speak.