Allocating AKS Nodes fails using Hub & Spoke vnet with Custom DNS

Question

I'm trying to implement the AKS cluster configuration outlined in this article: https://learn.microsoft.com/en-us/azure/aks/private-clusters#hub-and-spoke-with-custom-dns

However, the nodes in the vmss pool are not coming up due to not being able to resolve the cluster address listed in the private DNS. It seems to be ignoring the Private DNS Zone and only using the Hub DNS resolution (174.24.162.201). This is what the error looks like:

server can't find pocfedakseus2-898f96dd.7b763206-6384-4cab-a270-4550ae446b4a.privatelink.eastus2.azmk8s.io: NXDOMAIN\n\nServer:\t\t174.24.162.201\nAddress:\t174.24.162.201#53\n\n

I'm also looking at this note on the above at that article and it's not making sense to me:
If you are using Bring Your Own Route Table with kubenet and Bring Your Own DNS with Private Cluster, the cluster creation will fail. You will need to associate the RouteTable in the node resource group to the subnet after the cluster creation failed, in order to make the creation successful.

I am bringing my own route table with kubenet but if I try to deploy AKS without the route table on the subnet, that fails also.

Any ideas?

Answer 1

@Steven Ward
Apologies for the delay.

1. Can you please confirm if you are using custom DNS for the VNet?
2. Please share the details on the current dns setting used, and if you have a custom dns server, any attempt tried for the conditional forwarding?

Private DNS zone only works when attached to the vnet, and the vnet using Azure DNS. If you are using custom DNS setting by any chance, the DNS server need to forward DNS query to the "bind-conditional-forwarder", which is inside a vnet with private dns zone and azure dns, according the diagram.

Let me know. Thanks.

Answer 2

@KarishmaTiwari-MSFT ,

Thanks for the response. Yes, the vnet that the AKS cluster is residing in is using a custom DNS. I didn't set it up and don't have the privileges to answer all of questions. I will check with the administrators as to how bind-conditional-forwarder is set. Thank you

Answer 3

@KarishmaTiwari-MSFT
I was able to resolve the issue yesterday afternoon. Thank you for your insight and help.

We are using a custom DNS in both our hub vnet as well as the spoke vnet where the AKS cluster nodes were residing. Neither of the custom DNS servers included the Azure Internal DNS server (168.63.129.16). Because of that the nodes were not able to resolve an address specified in the cluster's private DNS zone (the API endpoint) and they were never successfully added to the cluster's node pool.

I added the Azure Internal DNS server (168.63.129.16) to the DNS server list for the spoke vnet. The nodes and the cluster are successfully provisioned now.

Thanks again.