AKS cluster stop deleting load balancer event

Question

Hey,

After upgrading AKS to version 21.2 I noticed in load balancer service events that azure-cloud-provider is trying to delete it, since there is a block on the resource group it failed. Is there any way to stop this process?

Answer 1

Hi @Veronika Bulavina

Thank you for reaching out to Microsoft Q&A forums,

AKS is a fully managed service , It would try to create all the backend required services when necessary and will also try to update/delete its resources when it feels that it is not necessary .
This should not impact your application , This is a generally performed to ensure a cost effective reliable service .

As a user of the service we should not apply any locks/blocks on the MC_resource group where all of these system operations would be performed .
AKS will try to retry the operation until it completes , So you might see multiple failed operations .

Hope this clarifies your question .

Thanks & Regards,
Pradeep

-----------------------------------------------------------

Please don't forget to accept the answer if it provides you clarity on the ask.

Answer 2

@Veronika Bulavina , adding on to @Pradeep Kommaraju 's answer,

The reason is a little misleading in this Service event. DeleteLoadBalancer operation from the Azure cloud controller manager actually refers to the cloud controller manager trying to remove the dynamic Public IP address associated with a Service of type LoadBalancer (public) when the Service is set to be deleted. This is default behaviour with Service objects using dynamic public IP addresses in Azure as mentioned here:

By default, the public IP address assigned to a load balancer resource created by an AKS cluster is only valid for the lifespan of that resource. If you delete the Kubernetes service, the associated load balancer and IP address are also deleted. If you want to assign a specific IP address or retain an IP address for redeployed Kubernetes services, you can create and use a static public IP address.

This is because, by default the following finalizer gets added to the Service object if no static IP address is configured to be used:

finalizers:  
  - service.kubernetes.io/load-balancer-cleanup  

You can verify this using:

kubectl get svc <service-name> -n <namespace> -o yaml  

If you add a static IP address with service.spec.loadBalancerIP this finalizer is not added to the Service object.

Thus, if the lock on the node resource group (default: MC_resourcegroupname_clustername_location) of the AKS cluster, is an absolute necessity for your use case, you might want to use Static public IP address with the Azure Kubernetes Service (AKS) load balancer. This will ensure that the Service object is deleted and the Public IP address is released but not removed, thus honoring your Delete lock on the node resource group.

----
P.S: If you are currently stuck with a Service that is being held up in terminating state due to this, please perform the following actions:

• Run kubectl edit svc <service-name> -n <namespace>
• Note the service.status.loadBalancer.ingress[0].ip value
• Remove the following block from the yaml:

       finalizers:  
          - service.kubernetes.io/load-balancer-cleanup  
  

• At this point the Service should get deleted.
• Remove the lock from the node resource group temporarily. In the node resource group go to the Load Balancer resource named kubernetes > Frontend IP configuration. Delete the frontend IP configuration (which is associated with the Public IP address noted from service.status.loadBalancer.ingress[0].ip value of the Service yaml) Illustration below:
• Go to All Services > Networking > Public IP addresses. Set filters for resource group (node resource group) and IP address as noted from service.status.loadBalancer.ingress[0].ip value of the Service yaml. Delete the public ip address resource.

----
Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.