While upgrading the AKS cluster version, we are getting the error as: Failed to save Kubernetes service 'aks-crushsftp-prd'. Error: Pre-allocated IPs 333 exceeds IPs available 246 in Subnet Cidr 10.56.27.0/24, Subnet Name ciosftp-prd-eastu2-aks.

Question

While upgrading the AKS cluster version, we are getting the error as: Failed to save Kubernetes service 'aks-crushsftp-prd'. Error: Pre-allocated IPs 333 exceeds IPs available 246 in Subnet Cidr 10.56.27.0/24, Subnet Name ciosftp-prd-eastu2-aks.

Answer 1

Hello Krishna satish Polasi Venkata

During the upgrade operation, buffer node(s) are added in order to run the workload of the node(s) that are being upgraded. After the upgrade is finished, the buffer node is removed. You can read more here: https://learn.microsoft.com/en-us/azure/aks/upgrade-cluster?tabs=azure-cli#upgrade-an-aks-cluster

The number of buffer node(s) are determined based on the node surge you specified. At this moment, if you don't specify the surge value, it adds an extra node by default. This behavior will change in Kubernetes version 1.27 when the surge value will be 10%.

That node(s) will require IP addresses and the number of IP addresses depends on the maxPods parameter value you use.

That message is saying that you don't have enough IP addresses for the buffer node(s) needed during the upgrade operation.

In this situation, you can reduce the cluster nodes by scaling down to free-up IP addresses for the upgrade.

If scaling down isn't an option, and your virtual network CIDR has enough IP addresses, try to add a node pool that has a unique subnet:

Add a new user node pool in the virtual network on a larger subnet.

Switch the original node pool to a system node pool type.

Scale up the user node pool.

Scale down the original node pool.

This official documentation is intended to address the situation you are facing, so I would highly recommend you to follow it: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-code-subnetisfull-upgrade

---

I hope this is helpful. If any clarification needed, let me know and I will do my best to answer.

Please "Accept as Answer" and Upvote if it helped, so that it can help others in the community looking for help on similar topics.

Thank you!

Answer 2

The error message suggests that there are more pre-allocated IP addresses for the Kubernetes service 'aks-crushsftp-prd' than the number of available IP addresses in the subnet CIDR 10.56.27.0/24.

To resolve this issue, you can try one of the following solutions:

1. Increase the size of the subnet CIDR: You can increase the size of the subnet CIDR to accommodate more pre-allocated IP addresses for the Kubernetes service.
2. Decrease the number of pre-allocated IP addresses: You can decrease the number of pre-allocated IP addresses for the Kubernetes service 'aks-crushsftp-prd' to fit within the available IP addresses in the subnet CIDR 10.56.27.0/24.
3. Create a new subnet with a larger CIDR: You can create a new subnet with a larger CIDR and move the Kubernetes service 'aks-crushsftp-prd' to the new subnet.

Before making any changes, it is recommended to review your network topology and ensure that the changes will not affect any other resources that are using the subnet.