Troubleshoot connection issues to pods or services within an AKS cluster (internal traffic)

This article discusses how to troubleshoot connection issues to pods or services as internal traffic from within the same Microsoft Azure Kubernetes Services (AKS) cluster.

Prerequisites

  • The Kubernetes kubectl tool, or a similar tool to connect to the cluster. To install kubectl by using Azure CLI, run the az aks install-cli command.

  • The apt-get command-line tool for handling packages.

  • The Client URL (cURL) tool, or a similar command-line tool.

  • The Netcat (nc) command-line tool for TCP connections.

Troubleshooting checklist

Step 1: Set up the test pod and remote server port

Set up the test pod and make sure that the required port is open on the remote server. From within the source pod (or a test pod that's in the same namespace as the source pod), follow these steps:

  1. Start a test pod in the cluster by running the kubectl run command:

    kubectl run -it --rm aks-ssh --namespace <namespace> --image=debian:stable
    
  2. After you gain access to the pod, run the following apt-get commands to install the DNS Utils, cURL, and Netcat packages:

    apt-get update -y
    apt-get install dnsutils -y
    apt-get install curl -y
    apt-get install netcat-openbsd -y
    
  3. After the packages are installed, run the following cURL command to test the connectivity to the IP address of the pod:

    curl -Iv http://<pod-ip-address>:<port>
    
  4. Run the Netcat command to check whether the remote server opened the required port:

    nc -z -v <endpoint> <port>
    

Step 2: View operational information about pods, containers, the Kubernetes services, and endpoints

Using kubectl and cURL at the command line, follow these steps to check that everything works as expected:

  1. Verify that the destination pod is up and running:

    kubectl get pods -n <namespace-name>
    

    If the destination pod is working correctly, the pod status is shown as Running, and the pod is shown as READY.

    NAME           READY   STATUS    RESTARTS   AGE
    my-other-pod   1/1     Running   0          44m
    my-pod         1/1     Running   0          44m
    
  2. Search the pod logs for access errors:

    kubectl logs <pod-name> -n <namespace-name>
    
  3. Search the pod logs for an individual container in a multicontainer pod:

    kubectl logs <pod-name> -n <namespace-name> -c <container-name>
    
  4. If the application that's inside the pod restarts repeatedly, view pod logs of a previous container instance to get the exit messages:

    kubectl logs <pod-name> --previous                      
    

    For the multicontainer case, use the following command:

    kubectl logs <pod-name> -c <container-name> --previous  
    
  5. Check whether there are any network policies that might block the traffic:

    kubectl get networkpolicies -A
    

    You should see output that resembles the following table.

    NAMESPACE     NAME                 POD-SELECTOR             AGE
    kube-system   konnectivity-agent   app=konnectivity-agent   4d1h
    

    If you see any other network policy that's custom-created, check whether that policy is blocking access to or from the pods.

  6. Check whether you can reach the application from the service IP address. First, show details about the service resource, such as the external IP address and port, by running the kubectl get services command:

    kubectl get services -n <namespace-name>
    
    NAME         TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)        AGE
    my-service   LoadBalancer   10.0.21.43   20.119.121.232   80:31773/TCP   28s
    

    Then, run cURL by using the service IP address and port to check whether you can reach the application:

    curl -Iv http://20.119.121.232:80
    .
    .
    .
    < HTTP/1.1 200 OK
    HTTP/1.1 200 OK
    
  7. Get more verbose information about the service:

    kubectl describe services <service-name> -n <namespace-name>
    
  8. Check the pod's IP address:

    kubectl get pods -o wide  
    
    NAME            READY   STATUS        RESTARTS   AGE   IP            NODE                                
    my-pod          1/1     Running       0          12m   10.244.0.15   aks-agentpool-000000-vmss000000  
    
  9. Verify that the pod's IP address exists as an endpoint in the service:

    kubectl describe services my-cluster-ip-service
    
    Name:              my-cluster-ip-service
    Namespace:         default
    Selector:          app=my-pod
    Type:              ClusterIP
    IP Family Policy:  SingleStack
    IP Families:       IPv4
    IP:                10.0.174.133
    IPs:               10.0.174.133
    Port:              <unset>  80/TCP
    TargetPort:        80/TCP
    Endpoints:         10.244.0.15:80     # <--- Here
    
  10. Verify the endpoints directly:

    kubectl get endpoints
    
    NAME                      ENDPOINTS           AGE
    my-cluster-ip-service     10.244.0.15:80      14m
    
  11. If the connection to a service doesn't work, restart the kube-proxy and CoreDNS pods:

    kubectl delete pods -n kube-system -l component=kube-proxy
    kubectl delete pods -n kube-system -l k8s-app=kube-dns
    
  12. Verify that the node isn't overused:

    kubectl top nodes
    

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.