DNS doesn't remove not ready pod in AKS with Azure CNI enabled

Sapozhnikov, Pavel 1 Reputation point
2021-09-10T01:34:03.147+00:00

How does AKS make not ready pod unavailable to accept requests into it? It only works if you have a service in front of that deployment correct?

I'd like to start this off by trying to explain what I had noticed in aks that is not configured with azure cni and then go on to explain what I have been seeing in aks with azure cni enabled and why the differences.

In AKS without cni enabled if I execute a curl on url on a not ready pod behind a service like this curl -I some-pod.some-service.some-namespace.svc.cluster.local:8080 what I get in the response is unresolvable hostname or something like that. Which means in my understanding that DNS doesn't have this entry. This is how in normal way aks handles not ready pods to not receives requests.

In AKS with azure cni enabled if I execute the same request on a not ready pod it is able to resolve the hostname and able to send request into the pod. However, there's one caveat is that when I try to execute a request through external private ip of that service that request doesn't reach the not ready pod which that is to be expected and seems to work right. But again when I try to execute a request like I mentioned above curl -I some-pod.some-service.some-namespace.svc.cluster.local:8080 that works but it shouldn't. Why does DNS in the case of azure cni have that value?

Is there anything I can do to configure azure cni to behave more like a default behavior of AKS where a curl request like that either will not resolve that hostname or will refuse the connection or something?

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,925 questions
{count} votes

1 answer

Sort by: Most helpful
  1. SRIJIT-BOSE-MSFT 4,331 Reputation points Microsoft Employee
    2021-09-13T15:09:30.753+00:00

    @Sapozhnikov, Pavel , Thank you for your question.

    Assuming that not ready pod refer to pods with Readiness Probe failing. The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers. [Reference]

    However, the logic determining the readiness of the pod might or might not have anything to do with whether the pod can serve requests and depends completely on the user.

    For instance with a Pod having the following manifest:

    apiVersion: v1  
    kind: Pod  
    metadata:  
      labels:  
        test: readiness  
      name: readiness-pod  
    spec:  
      containers:  
      - name: readiness-container  
        image: nginx  
        readinessProbe:  
          exec:  
            command:  
            - cat  
            - /tmp/healthy  
          initialDelaySeconds: 5  
          periodSeconds: 5  
    

    readiness is decided based on the existence of the file /tmp/healthy irrespective of whether nginx serves the application. So on running the application and exposing it using a service readiness-svc on k run -:

    kubectl exec readiness-pod -- /bin/bash -c 'if [ -f /tmp/healthy ]; then echo "/tmp/healthy file is present";else echo "/tmp/healthy file is absent";fi'  
    /tmp/healthy file is absent  
    
    kubectl get pods -o wide  
    NAME            READY   STATUS    RESTARTS   AGE    IP            NODE                                NOMINATED NODE   READINESS GATES  
    readiness-pod   0/1     Running   0          11m    10.240.0.28   aks-nodepool1-29819654-vmss000000   <none>           <none>  
    source-pod      1/1     Running   0          6h8m   10.240.0.27   aks-nodepool1-29819654-vmss000000   <none>           <none>  
    
    kubectl describe svc readiness-svc  
    Name:              readiness-svc  
    Namespace:         default  
    Labels:            test=readiness  
    Annotations:       <none>  
    Selector:          test=readiness  
    Type:              ClusterIP  
    IP Family Policy:  SingleStack  
    IP Families:       IPv4  
    IP:                10.0.23.194  
    IPs:               10.0.23.194  
    Port:              <unset>  80/TCP  
    TargetPort:        80/TCP  
    Endpoints:  
    Session Affinity:  None  
    Events:            <none>  
    
    kubectl exec -it source-pod -- bash  
    root@source-pod:/# curl -I readiness-svc.default.svc.cluster.local:80  
    curl: (7) Failed to connect to readiness-svc.default.svc.cluster.local port 80: Connection refused  
    root@source-pod:/# curl -I 10-240-0-28.default.pod.cluster.local:80  
    HTTP/1.1 200 OK  
    Server: nginx/1.21.3  
    Date: Mon, 13 Sep 2021 14:50:17 GMT  
    Content-Type: text/html  
    Content-Length: 615  
    Last-Modified: Tue, 07 Sep 2021 15:21:03 GMT  
    Connection: keep-alive  
    ETag: "6137835f-267"  
    Accept-Ranges: bytes  
    

    Thus, we can see that when we try to connect from source-pod to the service readiness-svc.default.svc.cluster.local on port 80, connection is refused. This is because the kubelet did not find the /tmp/healthy file in the readiness-pod container to perform a cat operation, consequently marking the Pod readiness-pod not ready to serve traffic and removing it from the backend of the Service readiness-svc. However, the nginx server on the pod can still serve a web application and it will continue to do so if you connect directly to the pod.

    Readiness probe failures of containers do not remove the DNS records of Pods. The DNS records of a Pod shares its lifespan with the Pod itself.

    This behavior is characteristic of Kubernetes and does not change with network plugins. We have attempted to reproduce the issue and have observed same behavior with AKS clusters using kubenet and Azure CNI network plugins.

    ----------

    Hope this helps.

    Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

    0 comments No comments