Upgrading AKS cluster to use NAT Gateway Outbound Type Failure

Question

Hello,

I am trying to upgrade my AKS cluster to use NAT Gateway as the outbound type, however, I am encountering an error related to this:

(CreateOrUpdateLoadBalancerError) Create or update load balancer failed: Subscription: *; resource group: *; loadbalancer name: kubernetes. Cannot remove backend address pool aksOutboundBackendPool from load balancer since it is in use by virtual machine scale set *.
Code: CreateOrUpdateLoadBalancerError
Message: Create or update load balancer failed: Subscription: *; resource group: *; loadbalancer name: kubernetes. Cannot remove backend address pool aksOutboundBackendPool from load balancer since it is in use by virtual machine scale set *.
Exception Details:	(None) Cannot remove backend address pool aksOutboundBackendPool from load balancer since it is in use by virtual machine scale set *.
	Code: None
	Message: Cannot remove backend address pool aksOutboundBackendPool from load balancer since it is in use by virtual machine scale set *.

The AKS cluster shows that the NAT Gateway is being set as the outbound type, however, from the error I see that the load balancer has not been cleaned up properly. I tried removing any reference to the load balancer on the VMSS that is shown in the error by remvoing virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools, but this does not work. The only thing that worked in my tests is to recreate the node pool which is not a great solution.

Any thoughts?

Thanks

Answer 1

hi Danny Wall & thx for sharing urs issue here at Q&A portal,

I guess the error is pretty specific AKS is trying to remove aksOutboundBackendPool from the managed load balancer, but the VMSS NIC config still references that backend pool. So the NAT Gateway setting may have partially applied, but the old outbound LB plumbing wasn’t fully detached from the node pool. Manually editing the VMSS is risky and AKS may just reconcile it back anyway. AKS-managed resources are allergic to manual surgery.

The clean options are usually drain and recreate the affected node pool or create a new node pool, move workloads, then delete the old one. That matches what u saw in testing, sadly.

Before doing that, I’d verify the cluster/node pool is on a current AKS version and try a node image upgrade, but I wouldn’t expect it to always fix stale LB backend pool references.

https://learn.microsoft.com/azure/aks/nat-gateway

https://learn.microsoft.com/azure/aks/egress-outboundtype

If this is production and recreating node pools is painful, open an AKS support ticket with the cluster name, node resource group, VMSS name, LB name, and failed operation ID. Microsoft may be able to clean the stuck managed resource state backend-side.

rgds,

Alex

&

If my answer was helpful pls mark it and additional thx if u follow me at Q&A portal

and at my blog https://ctrlaltdel.blog/