Cannot reach API-server of private AKS cluster

Question

Hi!

I am experiencing timeouts on kubectl commands towards my private AKS cluster.

The Azure portal states that the cluster is healthy and that the nodepool is running with a single target node. The are however 0 ready nodes but the "Diagnose and solve problems" tab reports no errors.

I communicate with the cluster through a jumpbox which manages to resolve the private FQDN to the correct IP address using "nslookup <cluster-fqdn>" but "curl -Iv https://<cluster-fqdn>" times out.

I have checked that the NSG allows port 10250 and that the firewall allows access to ports 9000, 1194 and 123.

Any suggestions to other troubleshooting steps that I could try?

Thanks!

Answer 1

Hi John,

Thanks for reaching out to us on a Microsoft Q&A forum.

Based on your inputs shared above which you followed for troubleshooting attempts. Please try with below steps mentioned:

• The Jumpbox VM is used to connect to the private AKS Cluster and manage it via kubectl.
• The Private AKS Cluster needs a private endpoint to connect to its API server via Azure Private Link.

1. Connection issues to the API server can occur for many reasons, but the root cause is often related to an error with one of these items:
   • Network
   • Authentication
   • Authorization
2. Enter the following az aks show command in Azure CLI. This command gets the fully qualified domain name (FQDN) of your AKS cluster.

      az aks show --resource-group <cluster-resource-group> --name <cluster-name> --query fqdn
   

3. With the FQDN, check whether the API server is reachable from the client machine by using the name server lookup (nslookup), client URL (curl), and telnet commands:

      # Check if the DNS Resolution is working: 
      $ nslookup <cluster-fqdn> 
      # Then check if the API Server is reachable: 
      $ curl -Iv https://<cluster-fqdn> 
      $ telnet <cluster-fqdn> 443
   

4. If the AKS cluster is private, make sure you run the command from a virtual machine (VM) that can access the AKS cluster's Azure Virtual Network. See Options for connecting to the private cluster.
5. To install the latest version of kubectl, run the az aks install-cli command in Azure CLI.
6. Make sure the network security group that's associated with AKS nodes allows communication on TCP port 10250 within the AKS nodes.
7. Since your cluster is private and its control plane is in the AKS subnet, it can't be connected to the API server unless it's in a network that can connect to the AKS subnet. It's an expected behavior.
8. In this case, try to access the API server from a client in a network that can communicate with the AKS subnet. Additionally, verify network security groups (NSGs) or other appliances between networks aren't blocking packets.
9. Open the necessary ports, FQDNs, and IP addresses
10. An AKS cluster with a private endpoint to the control plane / API server hosted by an AKS-managed Azure subscription. The cluster can communicate with the API server exposed via a Private Link Service using a private endpoint.

Please follow below documents for further reference:

https://learn.microsoft.com/en-us/azure/aks/private-clusters?tabs=default-basic-networking%2Cazure-portal#options-for-connecting-to-the-private-cluster

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connectivity/troubleshoot-cluster-connection-issues-api-server

If you found it helpful, could you kindly click the “Accept Answer and upvote” on the post.

If you have any further queries, please let us know we are glad to help you.