Azure DNS is blocking the Kubernetes cluster traffic

Question

We are setting up a Kubernetes cluster within a Private virtual Network. All master nodes and worker nodes are in a same private subnet but in different Availability Zones and they are accessible through a Public Load balancer.

All Kubernetes service pods are up and running but in kubernetes core-dns and api-server pod logs we are seeing below errors related to Azure DNS:

Core DNS Logs:

   [INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
   CoreDNS-1.6.7
   linux/amd64, go1.13.6, da7f65b
   [ERROR] plugin/errors: 2 1898679200261753048.7155559289384615854. HINFO: read udp 
   192.168.16.4:37176->168.63.129.16:53: i/o timeout
    [ERROR] plugin/errors: 2 1898679200261753048.7155559289384615854. HINFO: read udp 
    192.168.16.4:41483->168.63.129.16:53: i/o timeout
    [ERROR] plugin/errors: 2 1898679200261753048.7155559289384615854. HINFO: read udp 
    192.168.16.4:46000->168.63.129.16:53: i/o timeout

API Server Logs:

     E0901 04:17:35.747411       1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing 
     or missing response from https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: Get 
     https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting 
     for connection (Client.Timeout exceeded while awaiting headers)
     I0901 04:17:38.490026       1 log.go:172] http: TLS handshake error from 168.63.129.16:53930: EOF
     E0901 04:17:40.750482       1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing 
     or missing response from https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: Get 
     https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting 
     for connection (Client.Timeout exceeded while awaiting headers)
    W0901 04:17:43.815365       1 lease.go:224] Resetting endpoints for master service "kubernetes" to 
    [190.160.2.4 190.160.2.5]
    I0901 04:17:44.493715       1 log.go:172] http: TLS handshake error from 168.63.129.16:53947: EOF

Security Group Rules that we added in private subnet for our Master and Worker Nodes :

Inbound rules:

  Port                0-65535
  Protocol            TCP
  Source              Any
  Destination         Any 
  Action              Allow

   Port                0-65535
   Protocol            UDP
   Source              Any
   Destination         Any 
   Action              Allow

Outbound Rules:

   Port                Any
   Protocol            Any
   Source              Any
   Destination         Any 
   Action              Allow

Could someone please help us in understanding and resolving these issues. We are stucked due to these issues from long time.

Thank you in advance !!

Answer

@Bhushan Thank you providing the information.

As mentioned here:

https://learn.microsoft.com/en-US/troubleshoot/azure/general/support-policy-containers#third-party-orchestrators

Container orchestrators are enabled to work with container deployments as mentioned above, such as Windows Server Containers and Azure Kubernetes Service (AKS). Microsoft provides support for the deployment of common orchestrators such as Docker Swarm, Kubernetes, DC or system, and several others on AKS ONLY. Microsoft does not currently offer phone or web-based technical support for the configuration or operation of these container orchestrators within AKS or any support for third-party container orchestrators outside of AKS.

Since this is currently not a supported scenario I would recommend you to to post your query on Kubernetes forum here so that you get expertise help in timely manner :)

https://kubernetes.io/community/

Hope it helps!!!

Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics

Share via

Azure DNS is blocking the Kubernetes cluster traffic

1 answer