Internal Load Balancer using Azure CNI stuck on pending?

Question

Hello,

I am learning AKS and having difficulty understanding how to get Azure CNI to work with an Internal Load Balancer. I followed the docs directions in setting up a cluster and adding deployments and services. If I use Kubenet, I can get the Internal Load Balancer working without issue. However, simply switching to Azure CNI does not work for me. The service is stuck in "pending" for hours, no failure or other errors (see attached image).

While I have tried several attempts, I am currently doing as little as possible and letting AKS do most of the work. For example, when I go into Portal to set up a new Cluster, I simply click, "Azure CNI" and let the service auto-populate the fields for networking underneath. Even with the defaults that AKS provides, the deployment is still stuck.

Here is the yaml file I am using for Kubectl apply. Again, this file works for Kubenet, but gets stuck for Azure CNI.

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: aci-helloworld  
spec:  
  replicas: 1  
  selector:  
    matchLabels:  
      app: aci-helloworld  
  template:  
    metadata:  
      labels:  
        app: aci-helloworld  
    spec:  
      containers:  
      - name: aci-helloworld  
        image: mcr.microsoft.com/azuredocs/aci-helloworld  
        ports:  
        - containerPort: 80  
      nodeSelector:  
        beta.kubernetes.io/os: linux  
---  
apiVersion: v1  
kind: Service  
metadata:  
  name: aci-helloworld  
  annotations:  
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"  
spec:  
  type: LoadBalancer  
  ports:  
  - port: 80  
  selector:  
    app: aci-helloworld  
  

Guides I used:
https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni
https://learn.microsoft.com/en-us/azure/aks/internal-lb
https://learn.microsoft.com/en-us/azure/aks/ingress-internal-ip
https://learn.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal

Answer 1

Hello @rholliday ,
Thanks for your query . I just tried the same YAML file using kubectl create command and it works perfectly !

Couple of things to note for troubleshooting further:

1) Try to check the events : kubectl get events <service name> Or kubectl describe service <service Name>
eg: kubectl get events aci-helloworld
2) As it is internal Load balancer , you should see LB name as "kubernetes-internal " in the Azure Portal under node resource group (i.e. Resource group starts with name MC_)
3) Another way to troubleshoot is try to see the activity logs from Azure portal for that corresponding LB
4) Some times corresponding Service Principle might not have right authorization to VNETs/RGs
5) Or you might have hit IP address limit in your subscription.
6) Try also checking for kube-controller-manager events log section.

Additional articles to check the logs

• https://learn.microsoft.com/en-us/azure/aks/view-control-plane-logs

Try to check if Service Principle got expired

• https://dbafromthecold.com/2019/01/30/azure-kubernetes-service-error-loadbalancer-type-service-external-ip-stays-pending/

Let us know if those above troubleshooting helps out in find the issue.

If the above steps resolves your issue make sure to "Accept the answer" so that it will help the community out there.

Answer 2

@shiva patpi : the OP was able to resolve the issue using Service-Principal. But can you please help the rest of the folks who are still using SystemAssigned Managed Identity?? Can you please provide a solution for the rest of us??